2023-04-15 02:53:24,827 DEBUG [main] hbase.HBaseTestingUtility(342): Setting hbase.rootdir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/138098eb-2451-2962-d451-8fb43de970f3 2023-04-15 02:53:24,837 INFO [main] hbase.HBaseClassTestRule(94): Test class org.apache.hadoop.hbase.TestAcidGuaranteesWithEagerPolicy timeout: 13 mins 2023-04-15 02:53:24,851 INFO [Time-limited test] hbase.HBaseTestingUtility(1068): Starting up minicluster with option: StartMiniClusterOption{numMasters=1, masterClass=null, numRegionServers=1, rsPorts=, rsClass=null, numDataNodes=1, dataNodeHosts=null, numZkServers=1, createRootDir=false, createWALDir=false} 2023-04-15 02:53:24,851 INFO [Time-limited test] hbase.HBaseZKTestingUtility(82): Created new mini-cluster data directory: /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/138098eb-2451-2962-d451-8fb43de970f3/cluster_bb59e7f6-88db-fd92-3eb8-75ab1f712feb, deleteOnExit=true 2023-04-15 02:53:24,851 INFO [Time-limited test] hbase.HBaseTestingUtility(1082): STARTING DFS 2023-04-15 02:53:24,852 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting test.cache.data to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/138098eb-2451-2962-d451-8fb43de970f3/test.cache.data in system properties and HBase conf 2023-04-15 02:53:24,852 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting hadoop.tmp.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/138098eb-2451-2962-d451-8fb43de970f3/hadoop.tmp.dir in system properties and HBase conf 2023-04-15 02:53:24,852 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting hadoop.log.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/138098eb-2451-2962-d451-8fb43de970f3/hadoop.log.dir in system properties and HBase conf 2023-04-15 02:53:24,853 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting mapreduce.cluster.local.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/138098eb-2451-2962-d451-8fb43de970f3/mapreduce.cluster.local.dir in system properties and HBase conf 2023-04-15 02:53:24,853 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting mapreduce.cluster.temp.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/138098eb-2451-2962-d451-8fb43de970f3/mapreduce.cluster.temp.dir in system properties and HBase conf 2023-04-15 02:53:24,853 INFO [Time-limited test] hbase.HBaseTestingUtility(759): read short circuit is OFF 2023-04-15 02:53:24,951 WARN [Time-limited test] util.NativeCodeLoader(62): Unable to load native-hadoop library for your platform... using builtin-java classes where applicable 2023-04-15 02:53:25,262 DEBUG [Time-limited test] fs.HFileSystem(308): The file system is not a DistributedFileSystem. Skipping on block location reordering 2023-04-15 02:53:25,267 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.node-labels.fs-store.root-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/138098eb-2451-2962-d451-8fb43de970f3/yarn.node-labels.fs-store.root-dir in system properties and HBase conf 2023-04-15 02:53:25,267 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.node-attribute.fs-store.root-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/138098eb-2451-2962-d451-8fb43de970f3/yarn.node-attribute.fs-store.root-dir in system properties and HBase conf 2023-04-15 02:53:25,267 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.nodemanager.log-dirs to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/138098eb-2451-2962-d451-8fb43de970f3/yarn.nodemanager.log-dirs in system properties and HBase conf 2023-04-15 02:53:25,267 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.nodemanager.remote-app-log-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/138098eb-2451-2962-d451-8fb43de970f3/yarn.nodemanager.remote-app-log-dir in system properties and HBase conf 2023-04-15 02:53:25,268 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.timeline-service.entity-group-fs-store.active-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/138098eb-2451-2962-d451-8fb43de970f3/yarn.timeline-service.entity-group-fs-store.active-dir in system properties and HBase conf 2023-04-15 02:53:25,268 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.timeline-service.entity-group-fs-store.done-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/138098eb-2451-2962-d451-8fb43de970f3/yarn.timeline-service.entity-group-fs-store.done-dir in system properties and HBase conf 2023-04-15 02:53:25,268 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.nodemanager.remote-app-log-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/138098eb-2451-2962-d451-8fb43de970f3/yarn.nodemanager.remote-app-log-dir in system properties and HBase conf 2023-04-15 02:53:25,269 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting dfs.journalnode.edits.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/138098eb-2451-2962-d451-8fb43de970f3/dfs.journalnode.edits.dir in system properties and HBase conf 2023-04-15 02:53:25,269 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting dfs.datanode.shared.file.descriptor.paths to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/138098eb-2451-2962-d451-8fb43de970f3/dfs.datanode.shared.file.descriptor.paths in system properties and HBase conf 2023-04-15 02:53:25,269 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting nfs.dump.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/138098eb-2451-2962-d451-8fb43de970f3/nfs.dump.dir in system properties and HBase conf 2023-04-15 02:53:25,270 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting java.io.tmpdir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/138098eb-2451-2962-d451-8fb43de970f3/java.io.tmpdir in system properties and HBase conf 2023-04-15 02:53:25,270 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting dfs.journalnode.edits.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/138098eb-2451-2962-d451-8fb43de970f3/dfs.journalnode.edits.dir in system properties and HBase conf 2023-04-15 02:53:25,271 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting dfs.provided.aliasmap.inmemory.leveldb.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/138098eb-2451-2962-d451-8fb43de970f3/dfs.provided.aliasmap.inmemory.leveldb.dir in system properties and HBase conf 2023-04-15 02:53:25,271 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting fs.s3a.committer.staging.tmp.path to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/138098eb-2451-2962-d451-8fb43de970f3/fs.s3a.committer.staging.tmp.path in system properties and HBase conf Formatting using clusterid: testClusterID 2023-04-15 02:53:25,688 WARN [Time-limited test] conf.Configuration(1701): No unit for dfs.heartbeat.interval(3) assuming SECONDS 2023-04-15 02:53:25,691 WARN [Time-limited test] conf.Configuration(1701): No unit for dfs.namenode.safemode.extension(0) assuming MILLISECONDS 2023-04-15 02:53:25,919 WARN [Time-limited test] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-namenode.properties,hadoop-metrics2.properties 2023-04-15 02:53:26,056 INFO [Time-limited test] log.Slf4jLog(67): Logging to org.slf4j.impl.Reload4jLoggerAdapter(org.mortbay.log) via org.mortbay.log.Slf4jLog 2023-04-15 02:53:26,069 WARN [Time-limited test] http.HttpRequestLog(97): Jetty request log can only be enabled using Log4j 2023-04-15 02:53:26,099 INFO [Time-limited test] log.Slf4jLog(67): jetty-6.1.26 2023-04-15 02:53:26,125 INFO [Time-limited test] log.Slf4jLog(67): Extract jar:file:/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/local-repository/org/apache/hadoop/hadoop-hdfs/2.10.0/hadoop-hdfs-2.10.0-tests.jar!/webapps/hdfs to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/138098eb-2451-2962-d451-8fb43de970f3/java.io.tmpdir/Jetty_localhost_localdomain_34255_hdfs____.njbx79/webapp 2023-04-15 02:53:26,235 INFO [Time-limited test] log.Slf4jLog(67): Started HttpServer2$SelectChannelConnectorWithSafeStartup@localhost.localdomain:34255 2023-04-15 02:53:26,263 WARN [Time-limited test] conf.Configuration(1701): No unit for dfs.heartbeat.interval(3) assuming SECONDS 2023-04-15 02:53:26,264 WARN [Time-limited test] conf.Configuration(1701): No unit for dfs.namenode.safemode.extension(0) assuming MILLISECONDS 2023-04-15 02:53:26,601 WARN [Listener at localhost.localdomain/34069] common.MetricsLoggerTask(153): Metrics logging will not be async since the logger is not log4j 2023-04-15 02:53:26,682 WARN [Listener at localhost.localdomain/34069] conf.Configuration(1701): No unit for dfs.datanode.outliers.report.interval(1800000) assuming MILLISECONDS 2023-04-15 02:53:26,697 WARN [Listener at localhost.localdomain/34069] http.HttpRequestLog(97): Jetty request log can only be enabled using Log4j 2023-04-15 02:53:26,704 INFO [Listener at localhost.localdomain/34069] log.Slf4jLog(67): jetty-6.1.26 2023-04-15 02:53:26,710 INFO [Listener at localhost.localdomain/34069] log.Slf4jLog(67): Extract jar:file:/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/local-repository/org/apache/hadoop/hadoop-hdfs/2.10.0/hadoop-hdfs-2.10.0-tests.jar!/webapps/datanode to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/138098eb-2451-2962-d451-8fb43de970f3/java.io.tmpdir/Jetty_localhost_38365_datanode____.5dzjl8/webapp 2023-04-15 02:53:26,790 INFO [Listener at localhost.localdomain/34069] log.Slf4jLog(67): Started HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:38365 2023-04-15 02:53:27,076 WARN [Listener at localhost.localdomain/33693] common.MetricsLoggerTask(153): Metrics logging will not be async since the logger is not log4j 2023-04-15 02:53:27,390 INFO [Block report processor] blockmanagement.BlockManager(2202): BLOCK* processReport 0x8999dd395eb1f787: Processing first storage report for DS-92562a97-a373-4aa6-a7a1-c095e020c194 from datanode 9228a71f-02f5-4551-b279-c9b574701634 2023-04-15 02:53:27,392 INFO [Block report processor] blockmanagement.BlockManager(2228): BLOCK* processReport 0x8999dd395eb1f787: from storage DS-92562a97-a373-4aa6-a7a1-c095e020c194 node DatanodeRegistration(127.0.0.1:45433, datanodeUuid=9228a71f-02f5-4551-b279-c9b574701634, infoPort=42669, infoSecurePort=0, ipcPort=33693, storageInfo=lv=-57;cid=testClusterID;nsid=1098120820;c=1681527205747), blocks: 0, hasStaleStorage: true, processing time: 1 msecs, invalidatedBlocks: 0 2023-04-15 02:53:27,392 INFO [Block report processor] blockmanagement.BlockManager(2202): BLOCK* processReport 0x8999dd395eb1f787: Processing first storage report for DS-5a862e4c-5b5d-447c-b883-cf575f5b84e8 from datanode 9228a71f-02f5-4551-b279-c9b574701634 2023-04-15 02:53:27,392 INFO [Block report processor] blockmanagement.BlockManager(2228): BLOCK* processReport 0x8999dd395eb1f787: from storage DS-5a862e4c-5b5d-447c-b883-cf575f5b84e8 node DatanodeRegistration(127.0.0.1:45433, datanodeUuid=9228a71f-02f5-4551-b279-c9b574701634, infoPort=42669, infoSecurePort=0, ipcPort=33693, storageInfo=lv=-57;cid=testClusterID;nsid=1098120820;c=1681527205747), blocks: 0, hasStaleStorage: false, processing time: 0 msecs, invalidatedBlocks: 0 2023-04-15 02:53:27,417 DEBUG [Listener at localhost.localdomain/33693] hbase.HBaseTestingUtility(649): Setting hbase.rootdir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/138098eb-2451-2962-d451-8fb43de970f3 2023-04-15 02:53:27,466 INFO [Listener at localhost.localdomain/33693] zookeeper.MiniZooKeeperCluster(258): Started connectionTimeout=30000, dir=/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/138098eb-2451-2962-d451-8fb43de970f3/cluster_bb59e7f6-88db-fd92-3eb8-75ab1f712feb/zookeeper_0, clientPort=51762, secureClientPort=-1, dataDir=/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/138098eb-2451-2962-d451-8fb43de970f3/cluster_bb59e7f6-88db-fd92-3eb8-75ab1f712feb/zookeeper_0/version-2, dataDirSize=424 dataLogDir=/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/138098eb-2451-2962-d451-8fb43de970f3/cluster_bb59e7f6-88db-fd92-3eb8-75ab1f712feb/zookeeper_0/version-2, dataLogSize=424 tickTime=2000, maxClientCnxns=300, minSessionTimeout=4000, maxSessionTimeout=40000, serverId=0 2023-04-15 02:53:27,479 INFO [Listener at localhost.localdomain/33693] zookeeper.MiniZooKeeperCluster(283): Started MiniZooKeeperCluster and ran 'stat' on client port=51762 2023-04-15 02:53:27,489 INFO [Listener at localhost.localdomain/33693] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-04-15 02:53:27,492 INFO [Listener at localhost.localdomain/33693] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-04-15 02:53:28,104 INFO [Listener at localhost.localdomain/33693] util.FSUtils(471): Created version file at hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5 with version=8 2023-04-15 02:53:28,104 INFO [Listener at localhost.localdomain/33693] hbase.HBaseTestingUtility(1406): Setting hbase.fs.tmp.dir to hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/hbase-staging 2023-04-15 02:53:28,348 INFO [Listener at localhost.localdomain/33693] metrics.MetricRegistriesLoader(60): Loaded MetricRegistries class org.apache.hadoop.hbase.metrics.impl.MetricRegistriesImpl 2023-04-15 02:53:28,695 INFO [Listener at localhost.localdomain/33693] client.ConnectionUtils(127): master/jenkins-hbase20:0 server-side Connection retries=45 2023-04-15 02:53:28,719 INFO [Listener at localhost.localdomain/33693] ipc.RpcExecutor(189): Instantiated default.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-04-15 02:53:28,720 INFO [Listener at localhost.localdomain/33693] ipc.RpcExecutor(189): Instantiated priority.RWQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=2, maxQueueLength=30, handlerCount=3 2023-04-15 02:53:28,720 INFO [Listener at localhost.localdomain/33693] ipc.RWQueueRpcExecutor(107): priority.RWQ.Fifo writeQueues=1 writeHandlers=1 readQueues=1 readHandlers=2 scanQueues=0 scanHandlers=0 2023-04-15 02:53:28,720 INFO [Listener at localhost.localdomain/33693] ipc.RpcExecutor(189): Instantiated replication.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-04-15 02:53:28,720 INFO [Listener at localhost.localdomain/33693] ipc.RpcExecutor(189): Instantiated metaPriority.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=1 2023-04-15 02:53:28,836 INFO [Listener at localhost.localdomain/33693] ipc.RpcServerFactory(64): Creating org.apache.hadoop.hbase.ipc.NettyRpcServer hosting hbase.pb.MasterService, hbase.pb.RegionServerStatusService, hbase.pb.LockService, hbase.pb.HbckService, hbase.pb.ClientMetaService, hbase.pb.ClientService, hbase.pb.AdminService 2023-04-15 02:53:28,895 DEBUG [Listener at localhost.localdomain/33693] util.ClassSize(228): Using Unsafe to estimate memory layout 2023-04-15 02:53:28,970 INFO [Listener at localhost.localdomain/33693] ipc.NettyRpcServer(120): Bind to /148.251.75.209:43619 2023-04-15 02:53:28,978 INFO [Listener at localhost.localdomain/33693] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-04-15 02:53:28,980 INFO [Listener at localhost.localdomain/33693] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-04-15 02:53:28,996 INFO [Listener at localhost.localdomain/33693] zookeeper.RecoverableZooKeeper(93): Process identifier=master:43619 connecting to ZooKeeper ensemble=127.0.0.1:51762 2023-04-15 02:53:29,026 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:436190x0, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2023-04-15 02:53:29,028 DEBUG [zk-event-processor-pool-0] zookeeper.ZKWatcher(623): master:43619-0x100b28b0e4a0000 connected 2023-04-15 02:53:29,055 DEBUG [Listener at localhost.localdomain/33693] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2023-04-15 02:53:29,056 DEBUG [Listener at localhost.localdomain/33693] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2023-04-15 02:53:29,060 DEBUG [Listener at localhost.localdomain/33693] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/acl 2023-04-15 02:53:29,070 DEBUG [Listener at localhost.localdomain/33693] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=default.FPBQ.Fifo, numCallQueues=1, port=43619 2023-04-15 02:53:29,071 DEBUG [Listener at localhost.localdomain/33693] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=priority.RWQ.Fifo.write, numCallQueues=1, port=43619 2023-04-15 02:53:29,072 DEBUG [Listener at localhost.localdomain/33693] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=priority.RWQ.Fifo.read, numCallQueues=1, port=43619 2023-04-15 02:53:29,072 DEBUG [Listener at localhost.localdomain/33693] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=replication.FPBQ.Fifo, numCallQueues=1, port=43619 2023-04-15 02:53:29,073 DEBUG [Listener at localhost.localdomain/33693] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=metaPriority.FPBQ.Fifo, numCallQueues=1, port=43619 2023-04-15 02:53:29,077 INFO [Listener at localhost.localdomain/33693] master.HMaster(444): hbase.rootdir=hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5, hbase.cluster.distributed=false 2023-04-15 02:53:29,134 INFO [Listener at localhost.localdomain/33693] client.ConnectionUtils(127): regionserver/jenkins-hbase20:0 server-side Connection retries=45 2023-04-15 02:53:29,134 INFO [Listener at localhost.localdomain/33693] ipc.RpcExecutor(189): Instantiated default.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-04-15 02:53:29,134 INFO [Listener at localhost.localdomain/33693] ipc.RpcExecutor(189): Instantiated priority.RWQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=2, maxQueueLength=30, handlerCount=3 2023-04-15 02:53:29,134 INFO [Listener at localhost.localdomain/33693] ipc.RWQueueRpcExecutor(107): priority.RWQ.Fifo writeQueues=1 writeHandlers=1 readQueues=1 readHandlers=2 scanQueues=0 scanHandlers=0 2023-04-15 02:53:29,135 INFO [Listener at localhost.localdomain/33693] ipc.RpcExecutor(189): Instantiated replication.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-04-15 02:53:29,135 INFO [Listener at localhost.localdomain/33693] ipc.RpcExecutor(189): Instantiated metaPriority.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=1 2023-04-15 02:53:29,139 INFO [Listener at localhost.localdomain/33693] ipc.RpcServerFactory(64): Creating org.apache.hadoop.hbase.ipc.NettyRpcServer hosting hbase.pb.ClientService, hbase.pb.AdminService 2023-04-15 02:53:29,141 INFO [Listener at localhost.localdomain/33693] ipc.NettyRpcServer(120): Bind to /148.251.75.209:34681 2023-04-15 02:53:29,143 INFO [Listener at localhost.localdomain/33693] hfile.BlockCacheFactory(142): Allocating BlockCache size=782.40 MB, blockSize=64 KB 2023-04-15 02:53:29,148 DEBUG [Listener at localhost.localdomain/33693] mob.MobFileCache(120): MobFileCache enabled with cacheSize=1000, evictPeriods=3600sec, evictRemainRatio=0.5 2023-04-15 02:53:29,149 INFO [Listener at localhost.localdomain/33693] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-04-15 02:53:29,151 INFO [Listener at localhost.localdomain/33693] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-04-15 02:53:29,153 INFO [Listener at localhost.localdomain/33693] zookeeper.RecoverableZooKeeper(93): Process identifier=regionserver:34681 connecting to ZooKeeper ensemble=127.0.0.1:51762 2023-04-15 02:53:29,157 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:346810x0, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2023-04-15 02:53:29,158 DEBUG [zk-event-processor-pool-0] zookeeper.ZKWatcher(623): regionserver:34681-0x100b28b0e4a0001 connected 2023-04-15 02:53:29,158 DEBUG [Listener at localhost.localdomain/33693] zookeeper.ZKUtil(164): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2023-04-15 02:53:29,159 DEBUG [Listener at localhost.localdomain/33693] zookeeper.ZKUtil(164): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2023-04-15 02:53:29,160 DEBUG [Listener at localhost.localdomain/33693] zookeeper.ZKUtil(164): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/acl 2023-04-15 02:53:29,160 DEBUG [Listener at localhost.localdomain/33693] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=default.FPBQ.Fifo, numCallQueues=1, port=34681 2023-04-15 02:53:29,161 DEBUG [Listener at localhost.localdomain/33693] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=priority.RWQ.Fifo.write, numCallQueues=1, port=34681 2023-04-15 02:53:29,161 DEBUG [Listener at localhost.localdomain/33693] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=priority.RWQ.Fifo.read, numCallQueues=1, port=34681 2023-04-15 02:53:29,161 DEBUG [Listener at localhost.localdomain/33693] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=replication.FPBQ.Fifo, numCallQueues=1, port=34681 2023-04-15 02:53:29,162 DEBUG [Listener at localhost.localdomain/33693] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=metaPriority.FPBQ.Fifo, numCallQueues=1, port=34681 2023-04-15 02:53:29,163 INFO [master/jenkins-hbase20:0:becomeActiveMaster] master.HMaster(2168): Adding backup master ZNode /hbase/backup-masters/jenkins-hbase20.apache.org,43619,1681527208212 2023-04-15 02:53:29,171 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/backup-masters 2023-04-15 02:53:29,173 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] zookeeper.ZKUtil(162): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on existing znode=/hbase/backup-masters/jenkins-hbase20.apache.org,43619,1681527208212 2023-04-15 02:53:29,189 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/master 2023-04-15 02:53:29,189 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/master 2023-04-15 02:53:29,189 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-04-15 02:53:29,190 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] zookeeper.ZKUtil(162): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on existing znode=/hbase/master 2023-04-15 02:53:29,191 INFO [master/jenkins-hbase20:0:becomeActiveMaster] master.ActiveMasterManager(227): Deleting ZNode for /hbase/backup-masters/jenkins-hbase20.apache.org,43619,1681527208212 from backup master directory 2023-04-15 02:53:29,191 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(162): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on existing znode=/hbase/master 2023-04-15 02:53:29,193 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/backup-masters/jenkins-hbase20.apache.org,43619,1681527208212 2023-04-15 02:53:29,193 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/backup-masters 2023-04-15 02:53:29,194 WARN [master/jenkins-hbase20:0:becomeActiveMaster] hbase.ZNodeClearer(69): Environment variable HBASE_ZNODE_FILE not set; znodes will not be cleared on crash by start scripts (Longer MTTR!) 2023-04-15 02:53:29,194 INFO [master/jenkins-hbase20:0:becomeActiveMaster] master.ActiveMasterManager(237): Registered as active master=jenkins-hbase20.apache.org,43619,1681527208212 2023-04-15 02:53:29,196 INFO [master/jenkins-hbase20:0:becomeActiveMaster] regionserver.ChunkCreator(497): Allocating data MemStoreChunkPool with chunk size 2 MB, max count 352, initial count 0 2023-04-15 02:53:29,197 INFO [master/jenkins-hbase20:0:becomeActiveMaster] regionserver.ChunkCreator(497): Allocating index MemStoreChunkPool with chunk size 204.80 KB, max count 391, initial count 0 2023-04-15 02:53:29,269 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] util.FSUtils(620): Created cluster ID file at hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/hbase.id with ID: 0c0cc570-e32c-4613-9fe3-5e3f3753acc5 2023-04-15 02:53:29,313 INFO [master/jenkins-hbase20:0:becomeActiveMaster] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-04-15 02:53:29,330 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-04-15 02:53:29,368 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] zookeeper.ReadOnlyZKClient(139): Connect 0x2a875802 to 127.0.0.1:51762 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-15 02:53:29,393 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@4c771351, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-15 02:53:29,411 INFO [master/jenkins-hbase20:0:becomeActiveMaster] region.MasterRegion(309): Create or load local region for table 'master:store', {NAME => 'proc', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-04-15 02:53:29,413 INFO [master/jenkins-hbase20:0:becomeActiveMaster] region.MasterRegionFlusherAndCompactor(132): Injected flushSize=134217728, flushPerChanges=1000000, flushIntervalMs=900000 2023-04-15 02:53:29,427 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] asyncfs.FanOutOneBlockAsyncDFSOutputHelper(264): ClientProtocol::create wrong number of arguments, should be hadoop 3.2 or below 2023-04-15 02:53:29,428 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] asyncfs.FanOutOneBlockAsyncDFSOutputHelper(270): ClientProtocol::create wrong number of arguments, should be hadoop 2.x 2023-04-15 02:53:29,429 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] asyncfs.FanOutOneBlockAsyncDFSOutputHelper(279): can not find SHOULD_REPLICATE flag, should be hadoop 2.x java.lang.IllegalArgumentException: No enum constant org.apache.hadoop.fs.CreateFlag.SHOULD_REPLICATE at java.lang.Enum.valueOf(Enum.java:238) at org.apache.hadoop.fs.CreateFlag.valueOf(CreateFlag.java:63) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.loadShouldReplicateFlag(FanOutOneBlockAsyncDFSOutputHelper.java:277) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.(FanOutOneBlockAsyncDFSOutputHelper.java:304) at java.lang.Class.forName0(Native Method) at java.lang.Class.forName(Class.java:264) at org.apache.hadoop.hbase.wal.AsyncFSWALProvider.load(AsyncFSWALProvider.java:139) at org.apache.hadoop.hbase.wal.WALFactory.getProviderClass(WALFactory.java:135) at org.apache.hadoop.hbase.wal.WALFactory.getProvider(WALFactory.java:175) at org.apache.hadoop.hbase.wal.WALFactory.(WALFactory.java:202) at org.apache.hadoop.hbase.wal.WALFactory.(WALFactory.java:182) at org.apache.hadoop.hbase.master.region.MasterRegion.create(MasterRegion.java:339) at org.apache.hadoop.hbase.master.region.MasterRegionFactory.create(MasterRegionFactory.java:104) at org.apache.hadoop.hbase.master.HMaster.finishActiveMasterInitialization(HMaster.java:855) at org.apache.hadoop.hbase.master.HMaster.startActiveMasterManager(HMaster.java:2193) at org.apache.hadoop.hbase.master.HMaster.lambda$run$0(HMaster.java:528) at java.lang.Thread.run(Thread.java:750) 2023-04-15 02:53:29,432 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(243): No decryptEncryptedDataEncryptionKey method in DFSClient, should be hadoop version with HDFS-12396 java.lang.NoSuchMethodException: org.apache.hadoop.hdfs.DFSClient.decryptEncryptedDataEncryptionKey(org.apache.hadoop.fs.FileEncryptionInfo) at java.lang.Class.getDeclaredMethod(Class.java:2130) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper.createTransparentCryptoHelperWithoutHDFS12396(FanOutOneBlockAsyncDFSOutputSaslHelper.java:182) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper.createTransparentCryptoHelper(FanOutOneBlockAsyncDFSOutputSaslHelper.java:241) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper.(FanOutOneBlockAsyncDFSOutputSaslHelper.java:252) at java.lang.Class.forName0(Native Method) at java.lang.Class.forName(Class.java:264) at org.apache.hadoop.hbase.wal.AsyncFSWALProvider.load(AsyncFSWALProvider.java:140) at org.apache.hadoop.hbase.wal.WALFactory.getProviderClass(WALFactory.java:135) at org.apache.hadoop.hbase.wal.WALFactory.getProvider(WALFactory.java:175) at org.apache.hadoop.hbase.wal.WALFactory.(WALFactory.java:202) at org.apache.hadoop.hbase.wal.WALFactory.(WALFactory.java:182) at org.apache.hadoop.hbase.master.region.MasterRegion.create(MasterRegion.java:339) at org.apache.hadoop.hbase.master.region.MasterRegionFactory.create(MasterRegionFactory.java:104) at org.apache.hadoop.hbase.master.HMaster.finishActiveMasterInitialization(HMaster.java:855) at org.apache.hadoop.hbase.master.HMaster.startActiveMasterManager(HMaster.java:2193) at org.apache.hadoop.hbase.master.HMaster.lambda$run$0(HMaster.java:528) at java.lang.Thread.run(Thread.java:750) 2023-04-15 02:53:29,433 INFO [master/jenkins-hbase20:0:becomeActiveMaster] wal.WALFactory(158): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.AsyncFSWALProvider 2023-04-15 02:53:29,459 INFO [master/jenkins-hbase20:0:becomeActiveMaster] regionserver.HRegion(7693): Creating {ENCODED => 1595e783b53d99cd5eef43b6debb2682, NAME => 'master:store,,1.1595e783b53d99cd5eef43b6debb2682.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='master:store', {NAME => 'proc', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, under table dir hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/MasterData/data/master/store-tmp 2023-04-15 02:53:29,483 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] regionserver.HRegion(866): Instantiated master:store,,1.1595e783b53d99cd5eef43b6debb2682.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-15 02:53:29,484 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] regionserver.HRegion(1604): Closing 1595e783b53d99cd5eef43b6debb2682, disabling compactions & flushes 2023-04-15 02:53:29,484 INFO [master/jenkins-hbase20:0:becomeActiveMaster] regionserver.HRegion(1626): Closing region master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-04-15 02:53:29,484 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] regionserver.HRegion(1647): Waiting without time limit for close lock on master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-04-15 02:53:29,484 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] regionserver.HRegion(1714): Acquired close lock on master:store,,1.1595e783b53d99cd5eef43b6debb2682. after waiting 0 ms 2023-04-15 02:53:29,484 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] regionserver.HRegion(1724): Updates disabled for region master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-04-15 02:53:29,484 INFO [master/jenkins-hbase20:0:becomeActiveMaster] regionserver.HRegion(1838): Closed master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-04-15 02:53:29,484 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] regionserver.HRegion(1558): Region close journal for 1595e783b53d99cd5eef43b6debb2682: 2023-04-15 02:53:29,486 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] region.MasterRegion(191): WALDir=hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/MasterData/WALs/jenkins-hbase20.apache.org,43619,1681527208212 2023-04-15 02:53:29,509 INFO [master/jenkins-hbase20:0:becomeActiveMaster] wal.AbstractFSWAL(489): WAL configuration: blocksize=256 MB, rollsize=128 MB, prefix=jenkins-hbase20.apache.org%2C43619%2C1681527208212, suffix=, logDir=hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/MasterData/WALs/jenkins-hbase20.apache.org,43619,1681527208212, archiveDir=hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/MasterData/oldWALs, maxLogs=10 2023-04-15 02:53:29,553 DEBUG [RS-EventLoopGroup-3-2] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:45433,DS-92562a97-a373-4aa6-a7a1-c095e020c194,DISK] 2023-04-15 02:53:29,560 DEBUG [RS-EventLoopGroup-3-2] asyncfs.ProtobufDecoder(123): Hadoop 3.2 and below use unshaded protobuf. java.lang.ClassNotFoundException: org.apache.hadoop.thirdparty.protobuf.MessageLite at java.net.URLClassLoader.findClass(URLClassLoader.java:387) at java.lang.ClassLoader.loadClass(ClassLoader.java:418) at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:352) at java.lang.ClassLoader.loadClass(ClassLoader.java:351) at java.lang.Class.forName0(Native Method) at java.lang.Class.forName(Class.java:264) at org.apache.hadoop.hbase.io.asyncfs.ProtobufDecoder.(ProtobufDecoder.java:118) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.processWriteBlockResponse(FanOutOneBlockAsyncDFSOutputHelper.java:340) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.access$100(FanOutOneBlockAsyncDFSOutputHelper.java:112) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper$4.operationComplete(FanOutOneBlockAsyncDFSOutputHelper.java:424) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:590) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:557) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:492) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.addListener(DefaultPromise.java:185) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.initialize(FanOutOneBlockAsyncDFSOutputHelper.java:418) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.access$300(FanOutOneBlockAsyncDFSOutputHelper.java:112) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper$5.operationComplete(FanOutOneBlockAsyncDFSOutputHelper.java:476) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper$5.operationComplete(FanOutOneBlockAsyncDFSOutputHelper.java:471) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:590) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListeners0(DefaultPromise.java:583) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:559) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:492) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.setValue0(DefaultPromise.java:636) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.setSuccess0(DefaultPromise.java:625) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.trySuccess(DefaultPromise.java:105) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPromise.trySuccess(DefaultChannelPromise.java:84) at org.apache.hbase.thirdparty.io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.fulfillConnectPromise(AbstractEpollChannel.java:653) at org.apache.hbase.thirdparty.io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.finishConnect(AbstractEpollChannel.java:691) at org.apache.hbase.thirdparty.io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.epollOutReady(AbstractEpollChannel.java:567) at org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:489) at org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:397) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-15 02:53:29,617 INFO [master/jenkins-hbase20:0:becomeActiveMaster] wal.AbstractFSWAL(806): New WAL /user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/MasterData/WALs/jenkins-hbase20.apache.org,43619,1681527208212/jenkins-hbase20.apache.org%2C43619%2C1681527208212.1681527209518 2023-04-15 02:53:29,618 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] wal.AbstractFSWAL(887): Create new AsyncFSWAL writer with pipeline: [DatanodeInfoWithStorage[127.0.0.1:45433,DS-92562a97-a373-4aa6-a7a1-c095e020c194,DISK]] 2023-04-15 02:53:29,619 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] regionserver.HRegion(7854): Opening region: {ENCODED => 1595e783b53d99cd5eef43b6debb2682, NAME => 'master:store,,1.1595e783b53d99cd5eef43b6debb2682.', STARTKEY => '', ENDKEY => ''} 2023-04-15 02:53:29,619 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] regionserver.HRegion(866): Instantiated master:store,,1.1595e783b53d99cd5eef43b6debb2682.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-15 02:53:29,621 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] regionserver.HRegion(7894): checking encryption for 1595e783b53d99cd5eef43b6debb2682 2023-04-15 02:53:29,622 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] regionserver.HRegion(7897): checking classloading for 1595e783b53d99cd5eef43b6debb2682 2023-04-15 02:53:29,669 INFO [StoreOpener-1595e783b53d99cd5eef43b6debb2682-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family proc of region 1595e783b53d99cd5eef43b6debb2682 2023-04-15 02:53:29,676 DEBUG [StoreOpener-1595e783b53d99cd5eef43b6debb2682-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/proc 2023-04-15 02:53:29,694 INFO [StoreOpener-1595e783b53d99cd5eef43b6debb2682-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1595e783b53d99cd5eef43b6debb2682 columnFamilyName proc 2023-04-15 02:53:29,707 INFO [StoreOpener-1595e783b53d99cd5eef43b6debb2682-1] regionserver.HStore(310): Store=1595e783b53d99cd5eef43b6debb2682/proc, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-15 02:53:29,713 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682 2023-04-15 02:53:29,715 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682 2023-04-15 02:53:29,729 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] regionserver.HRegion(1055): writing seq id for 1595e783b53d99cd5eef43b6debb2682 2023-04-15 02:53:29,733 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] wal.WALSplitUtil(408): Wrote file=hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-04-15 02:53:29,734 INFO [master/jenkins-hbase20:0:becomeActiveMaster] regionserver.HRegion(1072): Opened 1595e783b53d99cd5eef43b6debb2682; next sequenceid=2; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=66261337, jitterRate=-0.012629136443138123}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-04-15 02:53:29,734 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] regionserver.HRegion(965): Region open journal for 1595e783b53d99cd5eef43b6debb2682: 2023-04-15 02:53:29,736 INFO [master/jenkins-hbase20:0:becomeActiveMaster] region.MasterRegionFlusherAndCompactor(122): Constructor flushSize=134217728, flushPerChanges=1000000, flushIntervalMs=900000, compactMin=4 2023-04-15 02:53:29,752 INFO [master/jenkins-hbase20:0:becomeActiveMaster] region.RegionProcedureStore(104): Starting the Region Procedure Store, number threads=5 2023-04-15 02:53:29,752 INFO [master/jenkins-hbase20:0:becomeActiveMaster] procedure2.ProcedureExecutor(562): Starting 5 core workers (bigger of cpus/4 or 16) with max (burst) worker count=50 2023-04-15 02:53:29,754 INFO [master/jenkins-hbase20:0:becomeActiveMaster] region.RegionProcedureStore(255): Starting Region Procedure Store lease recovery... 2023-04-15 02:53:29,755 INFO [master/jenkins-hbase20:0:becomeActiveMaster] procedure2.ProcedureExecutor(582): Recovered RegionProcedureStore lease in 1 msec 2023-04-15 02:53:29,780 INFO [master/jenkins-hbase20:0:becomeActiveMaster] procedure2.ProcedureExecutor(596): Loaded RegionProcedureStore in 24 msec 2023-04-15 02:53:29,781 INFO [master/jenkins-hbase20:0:becomeActiveMaster] procedure2.RemoteProcedureDispatcher(95): Instantiated, coreThreads=3 (allowCoreThreadTimeOut=true), queueMaxSize=32, operationDelay=150 2023-04-15 02:53:29,801 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] assignment.AssignmentManager(253): hbase:meta replica znodes: [] 2023-04-15 02:53:29,806 INFO [master/jenkins-hbase20:0:becomeActiveMaster] master.RegionServerTracker(124): Starting RegionServerTracker; 0 have existing ServerCrashProcedures, 0 possibly 'live' servers, and 0 'splitting'. 2023-04-15 02:53:29,826 INFO [master/jenkins-hbase20:0:becomeActiveMaster] balancer.BaseLoadBalancer(1082): slop=0.001, systemTablesOnMaster=false 2023-04-15 02:53:29,829 INFO [master/jenkins-hbase20:0:becomeActiveMaster] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=1000000, runMaxSteps=false, stepsPerRegion=800, maxRunningTime=30000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, ServerLocalityCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 0.0 etc. 2023-04-15 02:53:29,830 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/balancer 2023-04-15 02:53:29,835 INFO [master/jenkins-hbase20:0:becomeActiveMaster] normalizer.RegionNormalizerWorker(118): Normalizer rate limit set to unlimited 2023-04-15 02:53:29,838 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/normalizer 2023-04-15 02:53:29,841 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-04-15 02:53:29,842 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/switch/split 2023-04-15 02:53:29,842 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/switch/merge 2023-04-15 02:53:29,852 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/snapshot-cleanup 2023-04-15 02:53:29,857 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/running 2023-04-15 02:53:29,857 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/running 2023-04-15 02:53:29,857 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-04-15 02:53:29,858 INFO [master/jenkins-hbase20:0:becomeActiveMaster] master.HMaster(744): Active/primary master=jenkins-hbase20.apache.org,43619,1681527208212, sessionid=0x100b28b0e4a0000, setting cluster-up flag (Was=false) 2023-04-15 02:53:29,870 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-04-15 02:53:29,874 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] procedure.ZKProcedureUtil(251): Clearing all znodes /hbase/flush-table-proc/acquired, /hbase/flush-table-proc/reached, /hbase/flush-table-proc/abort 2023-04-15 02:53:29,875 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] procedure.ZKProcedureCoordinator(245): Starting controller for procedure member=jenkins-hbase20.apache.org,43619,1681527208212 2023-04-15 02:53:29,879 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-04-15 02:53:29,882 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] procedure.ZKProcedureUtil(251): Clearing all znodes /hbase/online-snapshot/acquired, /hbase/online-snapshot/reached, /hbase/online-snapshot/abort 2023-04-15 02:53:29,883 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] procedure.ZKProcedureCoordinator(245): Starting controller for procedure member=jenkins-hbase20.apache.org,43619,1681527208212 2023-04-15 02:53:29,885 WARN [master/jenkins-hbase20:0:becomeActiveMaster] snapshot.SnapshotManager(302): Couldn't delete working snapshot directory: hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/.hbase-snapshot/.tmp 2023-04-15 02:53:29,966 INFO [RS:0;jenkins-hbase20:34681] regionserver.HRegionServer(952): ClusterId : 0c0cc570-e32c-4613-9fe3-5e3f3753acc5 2023-04-15 02:53:29,970 DEBUG [RS:0;jenkins-hbase20:34681] procedure.RegionServerProcedureManagerHost(43): Procedure flush-table-proc initializing 2023-04-15 02:53:29,974 DEBUG [RS:0;jenkins-hbase20:34681] procedure.RegionServerProcedureManagerHost(45): Procedure flush-table-proc initialized 2023-04-15 02:53:29,974 DEBUG [RS:0;jenkins-hbase20:34681] procedure.RegionServerProcedureManagerHost(43): Procedure online-snapshot initializing 2023-04-15 02:53:29,977 DEBUG [RS:0;jenkins-hbase20:34681] procedure.RegionServerProcedureManagerHost(45): Procedure online-snapshot initialized 2023-04-15 02:53:29,978 DEBUG [RS:0;jenkins-hbase20:34681] zookeeper.ReadOnlyZKClient(139): Connect 0x7ac38bcd to 127.0.0.1:51762 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-15 02:53:29,984 DEBUG [RS:0;jenkins-hbase20:34681] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@74046ae5, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-15 02:53:29,984 DEBUG [RS:0;jenkins-hbase20:34681] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@703be0bd, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=jenkins-hbase20.apache.org/148.251.75.209:0 2023-04-15 02:53:29,991 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] procedure2.ProcedureExecutor(1029): Stored pid=1, state=RUNNABLE:INIT_META_WRITE_FS_LAYOUT; InitMetaProcedure table=hbase:meta 2023-04-15 02:53:30,001 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_OPEN_REGION-master/jenkins-hbase20:0, corePoolSize=5, maxPoolSize=5 2023-04-15 02:53:30,001 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_CLOSE_REGION-master/jenkins-hbase20:0, corePoolSize=5, maxPoolSize=5 2023-04-15 02:53:30,002 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_SERVER_OPERATIONS-master/jenkins-hbase20:0, corePoolSize=5, maxPoolSize=5 2023-04-15 02:53:30,002 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_META_SERVER_OPERATIONS-master/jenkins-hbase20:0, corePoolSize=5, maxPoolSize=5 2023-04-15 02:53:30,002 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=M_LOG_REPLAY_OPS-master/jenkins-hbase20:0, corePoolSize=10, maxPoolSize=10 2023-04-15 02:53:30,002 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase20:0, corePoolSize=1, maxPoolSize=1 2023-04-15 02:53:30,002 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_MERGE_OPERATIONS-master/jenkins-hbase20:0, corePoolSize=2, maxPoolSize=2 2023-04-15 02:53:30,002 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_TABLE_OPERATIONS-master/jenkins-hbase20:0, corePoolSize=1, maxPoolSize=1 2023-04-15 02:53:30,003 INFO [master/jenkins-hbase20:0:becomeActiveMaster] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.procedure2.CompletedProcedureCleaner; timeout=30000, timestamp=1681527240003 2023-04-15 02:53:30,005 INFO [master/jenkins-hbase20:0:becomeActiveMaster] cleaner.DirScanPool(70): log_cleaner Cleaner pool size is 1 2023-04-15 02:53:30,006 DEBUG [RS:0;jenkins-hbase20:34681] regionserver.ShutdownHook(81): Installed shutdown hook thread: Shutdownhook:RS:0;jenkins-hbase20:34681 2023-04-15 02:53:30,009 DEBUG [PEWorker-1] procedure.InitMetaProcedure(92): Execute pid=1, state=RUNNABLE:INIT_META_WRITE_FS_LAYOUT, locked=true; InitMetaProcedure table=hbase:meta 2023-04-15 02:53:30,009 INFO [PEWorker-1] procedure.InitMetaProcedure(71): BOOTSTRAP: creating hbase:meta region 2023-04-15 02:53:30,010 INFO [RS:0;jenkins-hbase20:34681] regionserver.RegionServerCoprocessorHost(66): System coprocessor loading is enabled 2023-04-15 02:53:30,010 INFO [RS:0;jenkins-hbase20:34681] regionserver.RegionServerCoprocessorHost(67): Table coprocessor loading is enabled 2023-04-15 02:53:30,010 DEBUG [RS:0;jenkins-hbase20:34681] regionserver.HRegionServer(1023): About to register with Master. 2023-04-15 02:53:30,012 INFO [RS:0;jenkins-hbase20:34681] regionserver.HRegionServer(2810): reportForDuty to master=jenkins-hbase20.apache.org,43619,1681527208212 with isa=jenkins-hbase20.apache.org/148.251.75.209:34681, startcode=1681527209133 2023-04-15 02:53:30,013 INFO [PEWorker-1] util.FSTableDescriptors(128): Creating new hbase:meta table descriptor 'hbase:meta', {TABLE_ATTRIBUTES => {IS_META => 'true', coprocessor$1 => '|org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint|536870911|'}}, {NAME => 'info', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '3', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'}, {NAME => 'rep_barrier', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '2147483647', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'table', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '3', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'} 2023-04-15 02:53:30,014 INFO [master/jenkins-hbase20:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveLogCleaner 2023-04-15 02:53:30,020 INFO [master/jenkins-hbase20:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.replication.master.ReplicationLogCleaner 2023-04-15 02:53:30,020 INFO [master/jenkins-hbase20:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveMasterLocalStoreWALCleaner 2023-04-15 02:53:30,020 INFO [master/jenkins-hbase20:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveProcedureWALCleaner 2023-04-15 02:53:30,021 INFO [master/jenkins-hbase20:0:becomeActiveMaster] cleaner.LogCleaner(148): Creating 1 old WALs cleaner threads 2023-04-15 02:53:30,021 INFO [master/jenkins-hbase20:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=LogsCleaner, period=600000, unit=MILLISECONDS is enabled. 2023-04-15 02:53:30,025 INFO [master/jenkins-hbase20:0:becomeActiveMaster] cleaner.DirScanPool(70): hfile_cleaner Cleaner pool size is 2 2023-04-15 02:53:30,026 INFO [master/jenkins-hbase20:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveMasterLocalStoreHFileCleaner 2023-04-15 02:53:30,026 INFO [master/jenkins-hbase20:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.HFileLinkCleaner 2023-04-15 02:53:30,029 INFO [master/jenkins-hbase20:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.snapshot.SnapshotHFileCleaner 2023-04-15 02:53:30,029 INFO [master/jenkins-hbase20:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveHFileCleaner 2023-04-15 02:53:30,031 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] cleaner.HFileCleaner(242): Starting for large file=Thread[master/jenkins-hbase20:0:becomeActiveMaster-HFileCleaner.large.0-1681527210030,5,FailOnTimeoutGroup] 2023-04-15 02:53:30,031 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] cleaner.HFileCleaner(257): Starting for small files=Thread[master/jenkins-hbase20:0:becomeActiveMaster-HFileCleaner.small.0-1681527210031,5,FailOnTimeoutGroup] 2023-04-15 02:53:30,031 INFO [master/jenkins-hbase20:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=HFileCleaner, period=600000, unit=MILLISECONDS is enabled. 2023-04-15 02:53:30,031 INFO [master/jenkins-hbase20:0:becomeActiveMaster] master.HMaster(1461): Reopening regions with very high storeFileRefCount is disabled. Provide threshold value > 0 for hbase.regions.recovery.store.file.ref.count to enable it. 2023-04-15 02:53:30,032 INFO [master/jenkins-hbase20:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=ReplicationBarrierCleaner, period=43200000, unit=MILLISECONDS is enabled. 2023-04-15 02:53:30,032 INFO [master/jenkins-hbase20:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=SnapshotCleaner, period=1800000, unit=MILLISECONDS is enabled. 2023-04-15 02:53:30,033 DEBUG [RS:0;jenkins-hbase20:34681] ipc.RpcConnection(124): Using SIMPLE authentication for service=RegionServerStatusService, sasl=false 2023-04-15 02:53:30,077 INFO [RS-EventLoopGroup-1-2] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:51555, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins.hfs.0 (auth:SIMPLE), service=RegionServerStatusService 2023-04-15 02:53:30,085 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.ServerManager(394): Registering regionserver=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:30,099 DEBUG [RS:0;jenkins-hbase20:34681] regionserver.HRegionServer(1596): Config from master: hbase.rootdir=hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5 2023-04-15 02:53:30,099 DEBUG [RS:0;jenkins-hbase20:34681] regionserver.HRegionServer(1596): Config from master: fs.defaultFS=hdfs://localhost.localdomain:34069 2023-04-15 02:53:30,099 DEBUG [RS:0;jenkins-hbase20:34681] regionserver.HRegionServer(1596): Config from master: hbase.master.info.port=-1 2023-04-15 02:53:30,103 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-04-15 02:53:30,103 DEBUG [RS:0;jenkins-hbase20:34681] zookeeper.ZKUtil(162): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:30,104 WARN [RS:0;jenkins-hbase20:34681] hbase.ZNodeClearer(69): Environment variable HBASE_ZNODE_FILE not set; znodes will not be cleared on crash by start scripts (Longer MTTR!) 2023-04-15 02:53:30,104 INFO [RS:0;jenkins-hbase20:34681] wal.WALFactory(158): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.AsyncFSWALProvider 2023-04-15 02:53:30,105 DEBUG [RS:0;jenkins-hbase20:34681] regionserver.HRegionServer(1947): logDir=hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/WALs/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:30,106 INFO [RegionServerTracker-0] master.RegionServerTracker(190): RegionServer ephemeral node created, adding [jenkins-hbase20.apache.org,34681,1681527209133] 2023-04-15 02:53:30,115 DEBUG [RS:0;jenkins-hbase20:34681] zookeeper.ZKUtil(162): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:30,126 DEBUG [RS:0;jenkins-hbase20:34681] regionserver.Replication(139): Replication stats-in-log period=300 seconds 2023-04-15 02:53:30,132 INFO [RS:0;jenkins-hbase20:34681] regionserver.MetricsRegionServerWrapperImpl(154): Computing regionserver metrics every 5000 milliseconds 2023-04-15 02:53:30,149 INFO [RS:0;jenkins-hbase20:34681] regionserver.MemStoreFlusher(125): globalMemStoreLimit=782.4 M, globalMemStoreLimitLowMark=743.3 M, Offheap=false 2023-04-15 02:53:30,152 INFO [RS:0;jenkins-hbase20:34681] throttle.PressureAwareCompactionThroughputController(131): Compaction throughput configurations, higher bound: 100.00 MB/second, lower bound 50.00 MB/second, off peak: unlimited, tuning period: 60000 ms 2023-04-15 02:53:30,152 INFO [RS:0;jenkins-hbase20:34681] hbase.ChoreService(166): Chore ScheduledChore name=CompactionThroughputTuner, period=60000, unit=MILLISECONDS is enabled. 2023-04-15 02:53:30,153 INFO [RS:0;jenkins-hbase20:34681] regionserver.HRegionServer$CompactionChecker(1838): CompactionChecker runs every PT1S 2023-04-15 02:53:30,159 INFO [RS:0;jenkins-hbase20:34681] hbase.ChoreService(166): Chore ScheduledChore name=CompactedHFilesCleaner, period=120000, unit=MILLISECONDS is enabled. 2023-04-15 02:53:30,159 DEBUG [RS:0;jenkins-hbase20:34681] executor.ExecutorService(93): Starting executor service name=RS_OPEN_REGION-regionserver/jenkins-hbase20:0, corePoolSize=1, maxPoolSize=1 2023-04-15 02:53:30,159 DEBUG [RS:0;jenkins-hbase20:34681] executor.ExecutorService(93): Starting executor service name=RS_OPEN_META-regionserver/jenkins-hbase20:0, corePoolSize=1, maxPoolSize=1 2023-04-15 02:53:30,159 DEBUG [RS:0;jenkins-hbase20:34681] executor.ExecutorService(93): Starting executor service name=RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase20:0, corePoolSize=1, maxPoolSize=1 2023-04-15 02:53:30,159 DEBUG [RS:0;jenkins-hbase20:34681] executor.ExecutorService(93): Starting executor service name=RS_CLOSE_REGION-regionserver/jenkins-hbase20:0, corePoolSize=1, maxPoolSize=1 2023-04-15 02:53:30,160 DEBUG [RS:0;jenkins-hbase20:34681] executor.ExecutorService(93): Starting executor service name=RS_CLOSE_META-regionserver/jenkins-hbase20:0, corePoolSize=1, maxPoolSize=1 2023-04-15 02:53:30,160 DEBUG [RS:0;jenkins-hbase20:34681] executor.ExecutorService(93): Starting executor service name=RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase20:0, corePoolSize=2, maxPoolSize=2 2023-04-15 02:53:30,160 DEBUG [RS:0;jenkins-hbase20:34681] executor.ExecutorService(93): Starting executor service name=RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase20:0, corePoolSize=1, maxPoolSize=1 2023-04-15 02:53:30,160 DEBUG [RS:0;jenkins-hbase20:34681] executor.ExecutorService(93): Starting executor service name=RS_REGION_REPLICA_FLUSH_OPS-regionserver/jenkins-hbase20:0, corePoolSize=1, maxPoolSize=1 2023-04-15 02:53:30,160 DEBUG [RS:0;jenkins-hbase20:34681] executor.ExecutorService(93): Starting executor service name=RS_REFRESH_PEER-regionserver/jenkins-hbase20:0, corePoolSize=1, maxPoolSize=1 2023-04-15 02:53:30,160 DEBUG [RS:0;jenkins-hbase20:34681] executor.ExecutorService(93): Starting executor service name=RS_SWITCH_RPC_THROTTLE-regionserver/jenkins-hbase20:0, corePoolSize=1, maxPoolSize=1 2023-04-15 02:53:30,161 INFO [RS:0;jenkins-hbase20:34681] hbase.ChoreService(166): Chore ScheduledChore name=CompactionChecker, period=1000, unit=MILLISECONDS is enabled. 2023-04-15 02:53:30,161 INFO [RS:0;jenkins-hbase20:34681] hbase.ChoreService(166): Chore ScheduledChore name=MemstoreFlusherChore, period=1000, unit=MILLISECONDS is enabled. 2023-04-15 02:53:30,161 INFO [RS:0;jenkins-hbase20:34681] hbase.ChoreService(166): Chore ScheduledChore name=nonceCleaner, period=360000, unit=MILLISECONDS is enabled. 2023-04-15 02:53:30,173 INFO [RS:0;jenkins-hbase20:34681] regionserver.HeapMemoryManager(209): Starting, tuneOn=false 2023-04-15 02:53:30,175 INFO [RS:0;jenkins-hbase20:34681] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase20.apache.org,34681,1681527209133-HeapMemoryTunerChore, period=60000, unit=MILLISECONDS is enabled. 2023-04-15 02:53:30,188 INFO [RS:0;jenkins-hbase20:34681] regionserver.Replication(203): jenkins-hbase20.apache.org,34681,1681527209133 started 2023-04-15 02:53:30,188 INFO [RS:0;jenkins-hbase20:34681] regionserver.HRegionServer(1638): Serving as jenkins-hbase20.apache.org,34681,1681527209133, RpcServer on jenkins-hbase20.apache.org/148.251.75.209:34681, sessionid=0x100b28b0e4a0001 2023-04-15 02:53:30,189 DEBUG [RS:0;jenkins-hbase20:34681] procedure.RegionServerProcedureManagerHost(51): Procedure flush-table-proc starting 2023-04-15 02:53:30,189 DEBUG [RS:0;jenkins-hbase20:34681] flush.RegionServerFlushTableProcedureManager(106): Start region server flush procedure manager jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:30,189 DEBUG [RS:0;jenkins-hbase20:34681] procedure.ZKProcedureMemberRpcs(357): Starting procedure member 'jenkins-hbase20.apache.org,34681,1681527209133' 2023-04-15 02:53:30,189 DEBUG [RS:0;jenkins-hbase20:34681] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-15 02:53:30,190 DEBUG [RS:0;jenkins-hbase20:34681] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-15 02:53:30,190 DEBUG [RS:0;jenkins-hbase20:34681] procedure.RegionServerProcedureManagerHost(53): Procedure flush-table-proc started 2023-04-15 02:53:30,190 DEBUG [RS:0;jenkins-hbase20:34681] procedure.RegionServerProcedureManagerHost(51): Procedure online-snapshot starting 2023-04-15 02:53:30,190 DEBUG [RS:0;jenkins-hbase20:34681] snapshot.RegionServerSnapshotManager(126): Start Snapshot Manager jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:30,190 DEBUG [RS:0;jenkins-hbase20:34681] procedure.ZKProcedureMemberRpcs(357): Starting procedure member 'jenkins-hbase20.apache.org,34681,1681527209133' 2023-04-15 02:53:30,190 DEBUG [RS:0;jenkins-hbase20:34681] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/online-snapshot/abort' 2023-04-15 02:53:30,191 DEBUG [RS:0;jenkins-hbase20:34681] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/online-snapshot/acquired' 2023-04-15 02:53:30,191 DEBUG [RS:0;jenkins-hbase20:34681] procedure.RegionServerProcedureManagerHost(53): Procedure online-snapshot started 2023-04-15 02:53:30,191 INFO [RS:0;jenkins-hbase20:34681] quotas.RegionServerRpcQuotaManager(63): Quota support disabled 2023-04-15 02:53:30,191 INFO [RS:0;jenkins-hbase20:34681] quotas.RegionServerSpaceQuotaManager(80): Quota support disabled, not starting space quota manager. 2023-04-15 02:53:30,307 INFO [RS:0;jenkins-hbase20:34681] wal.AbstractFSWAL(489): WAL configuration: blocksize=256 MB, rollsize=128 MB, prefix=jenkins-hbase20.apache.org%2C34681%2C1681527209133, suffix=, logDir=hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/WALs/jenkins-hbase20.apache.org,34681,1681527209133, archiveDir=hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/oldWALs, maxLogs=32 2023-04-15 02:53:30,321 DEBUG [RS-EventLoopGroup-3-2] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:45433,DS-92562a97-a373-4aa6-a7a1-c095e020c194,DISK] 2023-04-15 02:53:30,325 INFO [RS:0;jenkins-hbase20:34681] wal.AbstractFSWAL(806): New WAL /user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/WALs/jenkins-hbase20.apache.org,34681,1681527209133/jenkins-hbase20.apache.org%2C34681%2C1681527209133.1681527210309 2023-04-15 02:53:30,326 DEBUG [RS:0;jenkins-hbase20:34681] wal.AbstractFSWAL(887): Create new AsyncFSWAL writer with pipeline: [DatanodeInfoWithStorage[127.0.0.1:45433,DS-92562a97-a373-4aa6-a7a1-c095e020c194,DISK]] 2023-04-15 02:53:30,448 DEBUG [PEWorker-1] util.FSTableDescriptors(570): Wrote into hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/hbase/meta/.tabledesc/.tableinfo.0000000001 2023-04-15 02:53:30,449 INFO [PEWorker-1] util.FSTableDescriptors(135): Updated hbase:meta table descriptor to hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/hbase/meta/.tabledesc/.tableinfo.0000000001 2023-04-15 02:53:30,449 INFO [PEWorker-1] regionserver.HRegion(7675): creating {ENCODED => 1588230740, NAME => 'hbase:meta,,1', STARTKEY => '', ENDKEY => ''}, tableDescriptor='hbase:meta', {TABLE_ATTRIBUTES => {IS_META => 'true', coprocessor$1 => '|org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint|536870911|'}}, {NAME => 'info', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '3', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'}, {NAME => 'rep_barrier', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '2147483647', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'table', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '3', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5 2023-04-15 02:53:30,867 DEBUG [PEWorker-1] regionserver.HRegion(866): Instantiated hbase:meta,,1.1588230740; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-15 02:53:30,871 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family info of region 1588230740 2023-04-15 02:53:30,875 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/hbase/meta/1588230740/info 2023-04-15 02:53:30,876 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName info 2023-04-15 02:53:30,877 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/info, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-15 02:53:30,877 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family rep_barrier of region 1588230740 2023-04-15 02:53:30,881 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/hbase/meta/1588230740/rep_barrier 2023-04-15 02:53:30,881 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName rep_barrier 2023-04-15 02:53:30,882 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/rep_barrier, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-15 02:53:30,882 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family table of region 1588230740 2023-04-15 02:53:30,884 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/hbase/meta/1588230740/table 2023-04-15 02:53:30,885 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName table 2023-04-15 02:53:30,886 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/table, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-15 02:53:30,888 DEBUG [PEWorker-1] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/hbase/meta/1588230740 2023-04-15 02:53:30,889 DEBUG [PEWorker-1] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/hbase/meta/1588230740 2023-04-15 02:53:30,892 DEBUG [PEWorker-1] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table hbase:meta descriptor;using region.getMemStoreFlushHeapSize/# of families (16.0 M)) instead. 2023-04-15 02:53:30,894 DEBUG [PEWorker-1] regionserver.HRegion(1055): writing seq id for 1588230740 2023-04-15 02:53:30,898 DEBUG [PEWorker-1] wal.WALSplitUtil(408): Wrote file=hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/hbase/meta/1588230740/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-04-15 02:53:30,900 INFO [PEWorker-1] regionserver.HRegion(1072): Opened 1588230740; next sequenceid=2; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=69959241, jitterRate=0.0424739271402359}, FlushLargeStoresPolicy{flushSizeLowerBound=16777216} 2023-04-15 02:53:30,900 DEBUG [PEWorker-1] regionserver.HRegion(965): Region open journal for 1588230740: 2023-04-15 02:53:30,900 DEBUG [PEWorker-1] regionserver.HRegion(1604): Closing 1588230740, disabling compactions & flushes 2023-04-15 02:53:30,900 INFO [PEWorker-1] regionserver.HRegion(1626): Closing region hbase:meta,,1.1588230740 2023-04-15 02:53:30,900 DEBUG [PEWorker-1] regionserver.HRegion(1647): Waiting without time limit for close lock on hbase:meta,,1.1588230740 2023-04-15 02:53:30,900 DEBUG [PEWorker-1] regionserver.HRegion(1714): Acquired close lock on hbase:meta,,1.1588230740 after waiting 0 ms 2023-04-15 02:53:30,900 DEBUG [PEWorker-1] regionserver.HRegion(1724): Updates disabled for region hbase:meta,,1.1588230740 2023-04-15 02:53:30,901 INFO [PEWorker-1] regionserver.HRegion(1838): Closed hbase:meta,,1.1588230740 2023-04-15 02:53:30,902 DEBUG [PEWorker-1] regionserver.HRegion(1558): Region close journal for 1588230740: 2023-04-15 02:53:30,906 DEBUG [PEWorker-1] procedure.InitMetaProcedure(92): Execute pid=1, state=RUNNABLE:INIT_META_ASSIGN_META, locked=true; InitMetaProcedure table=hbase:meta 2023-04-15 02:53:30,906 INFO [PEWorker-1] procedure.InitMetaProcedure(103): Going to assign meta 2023-04-15 02:53:30,914 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=2, ppid=1, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=hbase:meta, region=1588230740, ASSIGN}] 2023-04-15 02:53:30,926 INFO [PEWorker-2] procedure.MasterProcedureScheduler(727): Took xlock for pid=2, ppid=1, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=hbase:meta, region=1588230740, ASSIGN 2023-04-15 02:53:30,929 INFO [PEWorker-2] assignment.TransitRegionStateProcedure(193): Starting pid=2, ppid=1, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=hbase:meta, region=1588230740, ASSIGN; state=OFFLINE, location=null; forceNewPlan=false, retain=false 2023-04-15 02:53:31,083 DEBUG [jenkins-hbase20:43619] assignment.AssignmentManager(2176): Processing assignQueue; systemServersCount=1, allServersCount=1 2023-04-15 02:53:31,089 INFO [PEWorker-3] zookeeper.MetaTableLocator(228): Setting hbase:meta replicaId=0 location in ZooKeeper as jenkins-hbase20.apache.org,34681,1681527209133, state=OPENING 2023-04-15 02:53:31,100 DEBUG [PEWorker-3] zookeeper.MetaTableLocator(240): hbase:meta region location doesn't exist, create it 2023-04-15 02:53:31,101 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-04-15 02:53:31,102 DEBUG [zk-event-processor-pool-0] master.MetaRegionLocationCache(164): Updating meta znode for path /hbase/meta-region-server: CHANGED 2023-04-15 02:53:31,104 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=3, ppid=2, state=RUNNABLE; OpenRegionProcedure 1588230740, server=jenkins-hbase20.apache.org,34681,1681527209133}] 2023-04-15 02:53:31,290 DEBUG [RSProcedureDispatcher-pool-0] master.ServerManager(712): New admin connection to jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:31,292 DEBUG [RSProcedureDispatcher-pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=AdminService, sasl=false 2023-04-15 02:53:31,295 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:38434, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=AdminService 2023-04-15 02:53:31,307 INFO [RS_OPEN_META-regionserver/jenkins-hbase20:0-0] handler.AssignRegionHandler(128): Open hbase:meta,,1.1588230740 2023-04-15 02:53:31,308 INFO [RS_OPEN_META-regionserver/jenkins-hbase20:0-0] wal.WALFactory(158): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.AsyncFSWALProvider 2023-04-15 02:53:31,311 INFO [RS_OPEN_META-regionserver/jenkins-hbase20:0-0] wal.AbstractFSWAL(489): WAL configuration: blocksize=256 MB, rollsize=128 MB, prefix=jenkins-hbase20.apache.org%2C34681%2C1681527209133.meta, suffix=.meta, logDir=hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/WALs/jenkins-hbase20.apache.org,34681,1681527209133, archiveDir=hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/oldWALs, maxLogs=32 2023-04-15 02:53:31,326 DEBUG [RS-EventLoopGroup-3-1] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:45433,DS-92562a97-a373-4aa6-a7a1-c095e020c194,DISK] 2023-04-15 02:53:31,330 INFO [RS_OPEN_META-regionserver/jenkins-hbase20:0-0] wal.AbstractFSWAL(806): New WAL /user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/WALs/jenkins-hbase20.apache.org,34681,1681527209133/jenkins-hbase20.apache.org%2C34681%2C1681527209133.meta.1681527211312.meta 2023-04-15 02:53:31,331 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase20:0-0] wal.AbstractFSWAL(887): Create new AsyncFSWAL writer with pipeline: [DatanodeInfoWithStorage[127.0.0.1:45433,DS-92562a97-a373-4aa6-a7a1-c095e020c194,DISK]] 2023-04-15 02:53:31,331 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => 1588230740, NAME => 'hbase:meta,,1', STARTKEY => '', ENDKEY => ''} 2023-04-15 02:53:31,333 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase20:0-0] coprocessor.CoprocessorHost(215): Loading coprocessor class org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint with path null and priority 536870911 2023-04-15 02:53:31,347 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(8550): Registered coprocessor service: region=hbase:meta,,1 service=MultiRowMutationService 2023-04-15 02:53:31,352 INFO [RS_OPEN_META-regionserver/jenkins-hbase20:0-0] regionserver.RegionCoprocessorHost(393): Loaded coprocessor org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint from HTD of hbase:meta successfully. 2023-04-15 02:53:31,357 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase20:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table meta 1588230740 2023-04-15 02:53:31,357 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(866): Instantiated hbase:meta,,1.1588230740; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-15 02:53:31,357 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7894): checking encryption for 1588230740 2023-04-15 02:53:31,357 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7897): checking classloading for 1588230740 2023-04-15 02:53:31,360 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family info of region 1588230740 2023-04-15 02:53:31,361 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/hbase/meta/1588230740/info 2023-04-15 02:53:31,361 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/hbase/meta/1588230740/info 2023-04-15 02:53:31,362 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName info 2023-04-15 02:53:31,363 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/info, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-15 02:53:31,363 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family rep_barrier of region 1588230740 2023-04-15 02:53:31,365 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/hbase/meta/1588230740/rep_barrier 2023-04-15 02:53:31,365 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/hbase/meta/1588230740/rep_barrier 2023-04-15 02:53:31,365 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName rep_barrier 2023-04-15 02:53:31,366 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/rep_barrier, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-15 02:53:31,366 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family table of region 1588230740 2023-04-15 02:53:31,368 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/hbase/meta/1588230740/table 2023-04-15 02:53:31,368 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/hbase/meta/1588230740/table 2023-04-15 02:53:31,368 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName table 2023-04-15 02:53:31,369 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/table, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-15 02:53:31,371 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/hbase/meta/1588230740 2023-04-15 02:53:31,373 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/hbase/meta/1588230740 2023-04-15 02:53:31,377 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase20:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table hbase:meta descriptor;using region.getMemStoreFlushHeapSize/# of families (16.0 M)) instead. 2023-04-15 02:53:31,379 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1055): writing seq id for 1588230740 2023-04-15 02:53:31,381 INFO [RS_OPEN_META-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1072): Opened 1588230740; next sequenceid=2; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=71777290, jitterRate=0.06956496834754944}, FlushLargeStoresPolicy{flushSizeLowerBound=16777216} 2023-04-15 02:53:31,381 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(965): Region open journal for 1588230740: 2023-04-15 02:53:31,388 INFO [RS_OPEN_META-regionserver/jenkins-hbase20:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for hbase:meta,,1.1588230740, pid=3, masterSystemTime=1681527211285 2023-04-15 02:53:31,403 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase20:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for hbase:meta,,1.1588230740 2023-04-15 02:53:31,404 INFO [RS_OPEN_META-regionserver/jenkins-hbase20:0-0] handler.AssignRegionHandler(156): Opened hbase:meta,,1.1588230740 2023-04-15 02:53:31,404 INFO [PEWorker-5] zookeeper.MetaTableLocator(228): Setting hbase:meta replicaId=0 location in ZooKeeper as jenkins-hbase20.apache.org,34681,1681527209133, state=OPEN 2023-04-15 02:53:31,407 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/meta-region-server 2023-04-15 02:53:31,407 DEBUG [zk-event-processor-pool-0] master.MetaRegionLocationCache(164): Updating meta znode for path /hbase/meta-region-server: CHANGED 2023-04-15 02:53:31,411 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=3, resume processing ppid=2 2023-04-15 02:53:31,411 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=3, ppid=2, state=SUCCESS; OpenRegionProcedure 1588230740, server=jenkins-hbase20.apache.org,34681,1681527209133 in 303 msec 2023-04-15 02:53:31,416 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=2, resume processing ppid=1 2023-04-15 02:53:31,416 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=2, ppid=1, state=SUCCESS; TransitRegionStateProcedure table=hbase:meta, region=1588230740, ASSIGN in 498 msec 2023-04-15 02:53:31,421 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=1, state=SUCCESS; InitMetaProcedure table=hbase:meta in 1.4950 sec 2023-04-15 02:53:31,421 INFO [master/jenkins-hbase20:0:becomeActiveMaster] master.HMaster(953): Wait for region servers to report in: status=null, state=RUNNING, startTime=1681527211421, completionTime=-1 2023-04-15 02:53:31,421 INFO [master/jenkins-hbase20:0:becomeActiveMaster] master.ServerManager(821): Finished waiting on RegionServer count=1; waited=0ms, expected min=1 server(s), max=1 server(s), master is running 2023-04-15 02:53:31,421 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] assignment.AssignmentManager(1517): Joining cluster... 2023-04-15 02:53:31,485 DEBUG [hconnection-0x48646271-shared-pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-15 02:53:31,489 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:38450, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-15 02:53:31,504 INFO [master/jenkins-hbase20:0:becomeActiveMaster] assignment.AssignmentManager(1529): Number of RegionServers=1 2023-04-15 02:53:31,504 INFO [master/jenkins-hbase20:0:becomeActiveMaster] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$RegionInTransitionChore; timeout=60000, timestamp=1681527271504 2023-04-15 02:53:31,504 INFO [master/jenkins-hbase20:0:becomeActiveMaster] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$DeadServerMetricRegionChore; timeout=120000, timestamp=1681527331504 2023-04-15 02:53:31,504 INFO [master/jenkins-hbase20:0:becomeActiveMaster] assignment.AssignmentManager(1536): Joined the cluster in 82 msec 2023-04-15 02:53:31,530 INFO [master/jenkins-hbase20:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase20.apache.org,43619,1681527208212-ClusterStatusChore, period=60000, unit=MILLISECONDS is enabled. 2023-04-15 02:53:31,530 INFO [master/jenkins-hbase20:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase20.apache.org,43619,1681527208212-BalancerChore, period=300000, unit=MILLISECONDS is enabled. 2023-04-15 02:53:31,530 INFO [master/jenkins-hbase20:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase20.apache.org,43619,1681527208212-RegionNormalizerChore, period=300000, unit=MILLISECONDS is enabled. 2023-04-15 02:53:31,531 INFO [master/jenkins-hbase20:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=CatalogJanitor-jenkins-hbase20:43619, period=300000, unit=MILLISECONDS is enabled. 2023-04-15 02:53:31,532 INFO [master/jenkins-hbase20:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=HbckChore-, period=3600000, unit=MILLISECONDS is enabled. 2023-04-15 02:53:31,537 DEBUG [master/jenkins-hbase20:0.Chore.1] janitor.CatalogJanitor(175): 2023-04-15 02:53:31,545 INFO [master/jenkins-hbase20:0:becomeActiveMaster] master.TableNamespaceManager(92): Namespace table not found. Creating... 2023-04-15 02:53:31,547 INFO [master/jenkins-hbase20:0:becomeActiveMaster] master.HMaster(2148): Client=null/null create 'hbase:namespace', {NAME => 'info', BLOOMFILTER => 'ROW', IN_MEMORY => 'true', VERSIONS => '10', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'} 2023-04-15 02:53:31,554 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] procedure2.ProcedureExecutor(1029): Stored pid=4, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION; CreateTableProcedure table=hbase:namespace 2023-04-15 02:53:31,556 INFO [PEWorker-4] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_PRE_OPERATION 2023-04-15 02:53:31,559 INFO [PEWorker-4] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_WRITE_FS_LAYOUT, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_WRITE_FS_LAYOUT 2023-04-15 02:53:31,578 DEBUG [HFileArchiver-1] backup.HFileArchiver(131): ARCHIVING hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/.tmp/data/hbase/namespace/5505718cae8473d3311d920897677a71 2023-04-15 02:53:31,581 DEBUG [HFileArchiver-1] backup.HFileArchiver(153): Directory hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/.tmp/data/hbase/namespace/5505718cae8473d3311d920897677a71 empty. 2023-04-15 02:53:31,582 DEBUG [HFileArchiver-1] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/.tmp/data/hbase/namespace/5505718cae8473d3311d920897677a71 2023-04-15 02:53:31,582 DEBUG [PEWorker-4] procedure.DeleteTableProcedure(328): Archived hbase:namespace regions 2023-04-15 02:53:32,015 DEBUG [PEWorker-4] util.FSTableDescriptors(570): Wrote into hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/.tmp/data/hbase/namespace/.tabledesc/.tableinfo.0000000001 2023-04-15 02:53:32,019 INFO [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(7675): creating {ENCODED => 5505718cae8473d3311d920897677a71, NAME => 'hbase:namespace,,1681527211546.5505718cae8473d3311d920897677a71.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='hbase:namespace', {NAME => 'info', BLOOMFILTER => 'ROW', IN_MEMORY => 'true', VERSIONS => '10', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/.tmp 2023-04-15 02:53:32,439 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(866): Instantiated hbase:namespace,,1681527211546.5505718cae8473d3311d920897677a71.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-15 02:53:32,440 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1604): Closing 5505718cae8473d3311d920897677a71, disabling compactions & flushes 2023-04-15 02:53:32,440 INFO [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1626): Closing region hbase:namespace,,1681527211546.5505718cae8473d3311d920897677a71. 2023-04-15 02:53:32,440 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1647): Waiting without time limit for close lock on hbase:namespace,,1681527211546.5505718cae8473d3311d920897677a71. 2023-04-15 02:53:32,441 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1714): Acquired close lock on hbase:namespace,,1681527211546.5505718cae8473d3311d920897677a71. after waiting 0 ms 2023-04-15 02:53:32,441 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1724): Updates disabled for region hbase:namespace,,1681527211546.5505718cae8473d3311d920897677a71. 2023-04-15 02:53:32,441 INFO [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1838): Closed hbase:namespace,,1681527211546.5505718cae8473d3311d920897677a71. 2023-04-15 02:53:32,441 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1558): Region close journal for 5505718cae8473d3311d920897677a71: 2023-04-15 02:53:32,448 INFO [PEWorker-4] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_ADD_TO_META, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_ADD_TO_META 2023-04-15 02:53:32,465 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"hbase:namespace,,1681527211546.5505718cae8473d3311d920897677a71.","families":{"info":[{"qualifier":"regioninfo","vlen":41,"tag":[],"timestamp":"1681527212451"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1681527212451"}]},"ts":"1681527212451"} 2023-04-15 02:53:32,488 INFO [PEWorker-4] hbase.MetaTableAccessor(1496): Added 1 regions to meta. 2023-04-15 02:53:32,490 INFO [PEWorker-4] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_ASSIGN_REGIONS, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_ASSIGN_REGIONS 2023-04-15 02:53:32,495 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"hbase:namespace","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1681527212490"}]},"ts":"1681527212490"} 2023-04-15 02:53:32,500 INFO [PEWorker-4] hbase.MetaTableAccessor(1635): Updated tableName=hbase:namespace, state=ENABLING in hbase:meta 2023-04-15 02:53:32,507 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=5, ppid=4, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=hbase:namespace, region=5505718cae8473d3311d920897677a71, ASSIGN}] 2023-04-15 02:53:32,511 INFO [PEWorker-3] procedure.MasterProcedureScheduler(727): Took xlock for pid=5, ppid=4, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=hbase:namespace, region=5505718cae8473d3311d920897677a71, ASSIGN 2023-04-15 02:53:32,513 INFO [PEWorker-3] assignment.TransitRegionStateProcedure(193): Starting pid=5, ppid=4, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=hbase:namespace, region=5505718cae8473d3311d920897677a71, ASSIGN; state=OFFLINE, location=jenkins-hbase20.apache.org,34681,1681527209133; forceNewPlan=false, retain=false 2023-04-15 02:53:32,665 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=5 updating hbase:meta row=5505718cae8473d3311d920897677a71, regionState=OPENING, regionLocation=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:32,665 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"hbase:namespace,,1681527211546.5505718cae8473d3311d920897677a71.","families":{"info":[{"qualifier":"regioninfo","vlen":41,"tag":[],"timestamp":"1681527212664"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1681527212664"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1681527212664"}]},"ts":"1681527212664"} 2023-04-15 02:53:32,675 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=6, ppid=5, state=RUNNABLE; OpenRegionProcedure 5505718cae8473d3311d920897677a71, server=jenkins-hbase20.apache.org,34681,1681527209133}] 2023-04-15 02:53:32,843 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase20:0-0] handler.AssignRegionHandler(128): Open hbase:namespace,,1681527211546.5505718cae8473d3311d920897677a71. 2023-04-15 02:53:32,845 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => 5505718cae8473d3311d920897677a71, NAME => 'hbase:namespace,,1681527211546.5505718cae8473d3311d920897677a71.', STARTKEY => '', ENDKEY => ''} 2023-04-15 02:53:32,845 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase20:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table namespace 5505718cae8473d3311d920897677a71 2023-04-15 02:53:32,845 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(866): Instantiated hbase:namespace,,1681527211546.5505718cae8473d3311d920897677a71.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-15 02:53:32,845 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7894): checking encryption for 5505718cae8473d3311d920897677a71 2023-04-15 02:53:32,845 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7897): checking classloading for 5505718cae8473d3311d920897677a71 2023-04-15 02:53:32,847 INFO [StoreOpener-5505718cae8473d3311d920897677a71-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family info of region 5505718cae8473d3311d920897677a71 2023-04-15 02:53:32,849 DEBUG [StoreOpener-5505718cae8473d3311d920897677a71-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/hbase/namespace/5505718cae8473d3311d920897677a71/info 2023-04-15 02:53:32,850 DEBUG [StoreOpener-5505718cae8473d3311d920897677a71-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/hbase/namespace/5505718cae8473d3311d920897677a71/info 2023-04-15 02:53:32,850 INFO [StoreOpener-5505718cae8473d3311d920897677a71-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 5505718cae8473d3311d920897677a71 columnFamilyName info 2023-04-15 02:53:32,851 INFO [StoreOpener-5505718cae8473d3311d920897677a71-1] regionserver.HStore(310): Store=5505718cae8473d3311d920897677a71/info, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-15 02:53:32,853 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/hbase/namespace/5505718cae8473d3311d920897677a71 2023-04-15 02:53:32,854 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/hbase/namespace/5505718cae8473d3311d920897677a71 2023-04-15 02:53:32,857 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1055): writing seq id for 5505718cae8473d3311d920897677a71 2023-04-15 02:53:32,862 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase20:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/hbase/namespace/5505718cae8473d3311d920897677a71/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-04-15 02:53:32,863 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1072): Opened 5505718cae8473d3311d920897677a71; next sequenceid=2; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=75042646, jitterRate=0.11822256445884705}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-04-15 02:53:32,863 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(965): Region open journal for 5505718cae8473d3311d920897677a71: 2023-04-15 02:53:32,864 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for hbase:namespace,,1681527211546.5505718cae8473d3311d920897677a71., pid=6, masterSystemTime=1681527212830 2023-04-15 02:53:32,868 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for hbase:namespace,,1681527211546.5505718cae8473d3311d920897677a71. 2023-04-15 02:53:32,868 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase20:0-0] handler.AssignRegionHandler(156): Opened hbase:namespace,,1681527211546.5505718cae8473d3311d920897677a71. 2023-04-15 02:53:32,869 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=5 updating hbase:meta row=5505718cae8473d3311d920897677a71, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:32,870 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"hbase:namespace,,1681527211546.5505718cae8473d3311d920897677a71.","families":{"info":[{"qualifier":"regioninfo","vlen":41,"tag":[],"timestamp":"1681527212869"},{"qualifier":"server","vlen":32,"tag":[],"timestamp":"1681527212869"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1681527212869"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1681527212869"}]},"ts":"1681527212869"} 2023-04-15 02:53:32,877 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=6, resume processing ppid=5 2023-04-15 02:53:32,877 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=6, ppid=5, state=SUCCESS; OpenRegionProcedure 5505718cae8473d3311d920897677a71, server=jenkins-hbase20.apache.org,34681,1681527209133 in 199 msec 2023-04-15 02:53:32,880 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=5, resume processing ppid=4 2023-04-15 02:53:32,881 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=5, ppid=4, state=SUCCESS; TransitRegionStateProcedure table=hbase:namespace, region=5505718cae8473d3311d920897677a71, ASSIGN in 370 msec 2023-04-15 02:53:32,882 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_UPDATE_DESC_CACHE, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_UPDATE_DESC_CACHE 2023-04-15 02:53:32,883 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"hbase:namespace","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1681527212882"}]},"ts":"1681527212882"} 2023-04-15 02:53:32,886 INFO [PEWorker-3] hbase.MetaTableAccessor(1635): Updated tableName=hbase:namespace, state=ENABLED in hbase:meta 2023-04-15 02:53:32,889 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_POST_OPERATION, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_POST_OPERATION 2023-04-15 02:53:32,891 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=4, state=SUCCESS; CreateTableProcedure table=hbase:namespace in 1.3420 sec 2023-04-15 02:53:32,959 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/namespace 2023-04-15 02:53:32,961 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/namespace 2023-04-15 02:53:32,961 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-04-15 02:53:33,007 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] procedure2.ProcedureExecutor(1029): Stored pid=7, state=RUNNABLE:CREATE_NAMESPACE_PREPARE; CreateNamespaceProcedure, namespace=default 2023-04-15 02:53:33,022 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/namespace 2023-04-15 02:53:33,029 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=7, state=SUCCESS; CreateNamespaceProcedure, namespace=default in 32 msec 2023-04-15 02:53:33,040 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] procedure2.ProcedureExecutor(1029): Stored pid=8, state=RUNNABLE:CREATE_NAMESPACE_PREPARE; CreateNamespaceProcedure, namespace=hbase 2023-04-15 02:53:33,052 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/namespace 2023-04-15 02:53:33,056 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=8, state=SUCCESS; CreateNamespaceProcedure, namespace=hbase in 15 msec 2023-04-15 02:53:33,074 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/namespace/default 2023-04-15 02:53:33,076 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/namespace/hbase 2023-04-15 02:53:33,077 INFO [master/jenkins-hbase20:0:becomeActiveMaster] master.HMaster(1083): Master has completed initialization 3.882sec 2023-04-15 02:53:33,079 INFO [master/jenkins-hbase20:0:becomeActiveMaster] quotas.MasterQuotaManager(97): Quota support disabled 2023-04-15 02:53:33,081 INFO [master/jenkins-hbase20:0:becomeActiveMaster] slowlog.SlowLogMasterService(57): Slow/Large requests logging to system table hbase:slowlog is disabled. Quitting. 2023-04-15 02:53:33,081 INFO [master/jenkins-hbase20:0:becomeActiveMaster] zookeeper.ZKWatcher(269): not a secure deployment, proceeding 2023-04-15 02:53:33,083 INFO [master/jenkins-hbase20:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase20.apache.org,43619,1681527208212-ExpiredMobFileCleanerChore, period=86400, unit=SECONDS is enabled. 2023-04-15 02:53:33,084 INFO [master/jenkins-hbase20:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase20.apache.org,43619,1681527208212-MobCompactionChore, period=604800, unit=SECONDS is enabled. 2023-04-15 02:53:33,096 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] master.HMaster(1175): Balancer post startup initialization complete, took 0 seconds 2023-04-15 02:53:33,179 DEBUG [Listener at localhost.localdomain/33693] zookeeper.ReadOnlyZKClient(139): Connect 0x056ecc29 to 127.0.0.1:51762 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-15 02:53:33,184 DEBUG [Listener at localhost.localdomain/33693] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@14fd5e70, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-15 02:53:33,195 DEBUG [hconnection-0x6e20e9aa-shared-pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-15 02:53:33,207 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:38466, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-15 02:53:33,215 INFO [Listener at localhost.localdomain/33693] hbase.HBaseTestingUtility(1145): Minicluster is up; activeMaster=jenkins-hbase20.apache.org,43619,1681527208212 2023-04-15 02:53:33,233 INFO [Listener at localhost.localdomain/33693] hbase.ResourceChecker(147): before: TestAcidGuaranteesWithEagerPolicy#testMixedAtomicity Thread=207, OpenFileDescriptor=470, MaxFileDescriptor=60000, SystemLoadAverage=413, ProcessCount=169, AvailableMemoryMB=6340 2023-04-15 02:53:33,249 DEBUG [Listener at localhost.localdomain/33693] ipc.RpcConnection(124): Using SIMPLE authentication for service=MasterService, sasl=false 2023-04-15 02:53:33,252 INFO [RS-EventLoopGroup-1-3] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:55170, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=MasterService 2023-04-15 02:53:33,260 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] util.TableDescriptorChecker(340): MEMSTORE_FLUSHSIZE for table descriptor or "hbase.hregion.memstore.flush.size" (131072) is too small, which might cause very frequent flushing. 2023-04-15 02:53:33,264 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.HMaster$4(2112): Client=jenkins//148.251.75.209 create 'TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-04-15 02:53:33,267 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] procedure2.ProcedureExecutor(1029): Stored pid=9, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION; CreateTableProcedure table=TestAcidGuarantees 2023-04-15 02:53:33,269 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_PRE_OPERATION 2023-04-15 02:53:33,271 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_WRITE_FS_LAYOUT, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_WRITE_FS_LAYOUT 2023-04-15 02:53:33,274 DEBUG [HFileArchiver-2] backup.HFileArchiver(131): ARCHIVING hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/.tmp/data/default/TestAcidGuarantees/c44270c3412c22e2c805f466aae6ff56 2023-04-15 02:53:33,275 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(697): Client=jenkins//148.251.75.209 procedure request for creating table: namespace: "default" qualifier: "TestAcidGuarantees" procId is: 9 2023-04-15 02:53:33,275 DEBUG [HFileArchiver-2] backup.HFileArchiver(153): Directory hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/.tmp/data/default/TestAcidGuarantees/c44270c3412c22e2c805f466aae6ff56 empty. 2023-04-15 02:53:33,276 DEBUG [HFileArchiver-2] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/.tmp/data/default/TestAcidGuarantees/c44270c3412c22e2c805f466aae6ff56 2023-04-15 02:53:33,276 DEBUG [PEWorker-2] procedure.DeleteTableProcedure(328): Archived TestAcidGuarantees regions 2023-04-15 02:53:33,287 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(1227): Checking to see if procedure is done pid=9 2023-04-15 02:53:33,394 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(1227): Checking to see if procedure is done pid=9 2023-04-15 02:53:33,598 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(1227): Checking to see if procedure is done pid=9 2023-04-15 02:53:33,701 DEBUG [PEWorker-2] util.FSTableDescriptors(570): Wrote into hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/.tmp/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000001 2023-04-15 02:53:33,705 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(7675): creating {ENCODED => c44270c3412c22e2c805f466aae6ff56, NAME => 'TestAcidGuarantees,,1681527213260.c44270c3412c22e2c805f466aae6ff56.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/.tmp 2023-04-15 02:53:33,903 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(1227): Checking to see if procedure is done pid=9 2023-04-15 02:53:34,120 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1681527213260.c44270c3412c22e2c805f466aae6ff56.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-15 02:53:34,122 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1604): Closing c44270c3412c22e2c805f466aae6ff56, disabling compactions & flushes 2023-04-15 02:53:34,122 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1681527213260.c44270c3412c22e2c805f466aae6ff56. 2023-04-15 02:53:34,122 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1681527213260.c44270c3412c22e2c805f466aae6ff56. 2023-04-15 02:53:34,122 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1681527213260.c44270c3412c22e2c805f466aae6ff56. after waiting 0 ms 2023-04-15 02:53:34,122 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1681527213260.c44270c3412c22e2c805f466aae6ff56. 2023-04-15 02:53:34,122 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1681527213260.c44270c3412c22e2c805f466aae6ff56. 2023-04-15 02:53:34,122 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1558): Region close journal for c44270c3412c22e2c805f466aae6ff56: 2023-04-15 02:53:34,128 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_ADD_TO_META, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_ADD_TO_META 2023-04-15 02:53:34,130 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1681527213260.c44270c3412c22e2c805f466aae6ff56.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1681527214129"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1681527214129"}]},"ts":"1681527214129"} 2023-04-15 02:53:34,132 INFO [PEWorker-2] hbase.MetaTableAccessor(1496): Added 1 regions to meta. 2023-04-15 02:53:34,134 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_ASSIGN_REGIONS, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_ASSIGN_REGIONS 2023-04-15 02:53:34,134 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1681527214134"}]},"ts":"1681527214134"} 2023-04-15 02:53:34,136 INFO [PEWorker-2] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=ENABLING in hbase:meta 2023-04-15 02:53:34,139 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=10, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=TestAcidGuarantees, region=c44270c3412c22e2c805f466aae6ff56, ASSIGN}] 2023-04-15 02:53:34,141 INFO [PEWorker-4] procedure.MasterProcedureScheduler(727): Took xlock for pid=10, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=TestAcidGuarantees, region=c44270c3412c22e2c805f466aae6ff56, ASSIGN 2023-04-15 02:53:34,142 INFO [PEWorker-4] assignment.TransitRegionStateProcedure(193): Starting pid=10, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=TestAcidGuarantees, region=c44270c3412c22e2c805f466aae6ff56, ASSIGN; state=OFFLINE, location=jenkins-hbase20.apache.org,34681,1681527209133; forceNewPlan=false, retain=false 2023-04-15 02:53:34,293 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=10 updating hbase:meta row=c44270c3412c22e2c805f466aae6ff56, regionState=OPENING, regionLocation=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:34,294 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1681527213260.c44270c3412c22e2c805f466aae6ff56.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1681527214293"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1681527214293"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1681527214293"}]},"ts":"1681527214293"} 2023-04-15 02:53:34,298 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=11, ppid=10, state=RUNNABLE; OpenRegionProcedure c44270c3412c22e2c805f466aae6ff56, server=jenkins-hbase20.apache.org,34681,1681527209133}] 2023-04-15 02:53:34,406 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(1227): Checking to see if procedure is done pid=9 2023-04-15 02:53:34,461 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] handler.AssignRegionHandler(128): Open TestAcidGuarantees,,1681527213260.c44270c3412c22e2c805f466aae6ff56. 2023-04-15 02:53:34,462 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => c44270c3412c22e2c805f466aae6ff56, NAME => 'TestAcidGuarantees,,1681527213260.c44270c3412c22e2c805f466aae6ff56.', STARTKEY => '', ENDKEY => ''} 2023-04-15 02:53:34,462 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table TestAcidGuarantees c44270c3412c22e2c805f466aae6ff56 2023-04-15 02:53:34,462 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1681527213260.c44270c3412c22e2c805f466aae6ff56.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-15 02:53:34,462 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7894): checking encryption for c44270c3412c22e2c805f466aae6ff56 2023-04-15 02:53:34,463 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7897): checking classloading for c44270c3412c22e2c805f466aae6ff56 2023-04-15 02:53:34,465 INFO [StoreOpener-c44270c3412c22e2c805f466aae6ff56-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family A of region c44270c3412c22e2c805f466aae6ff56 2023-04-15 02:53:34,468 DEBUG [StoreOpener-c44270c3412c22e2c805f466aae6ff56-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/c44270c3412c22e2c805f466aae6ff56/A 2023-04-15 02:53:34,468 DEBUG [StoreOpener-c44270c3412c22e2c805f466aae6ff56-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/c44270c3412c22e2c805f466aae6ff56/A 2023-04-15 02:53:34,470 INFO [StoreOpener-c44270c3412c22e2c805f466aae6ff56-1] regionserver.CompactingMemStore(121): Store=A, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-15 02:53:34,470 INFO [StoreOpener-c44270c3412c22e2c805f466aae6ff56-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region c44270c3412c22e2c805f466aae6ff56 columnFamilyName A 2023-04-15 02:53:34,471 INFO [StoreOpener-c44270c3412c22e2c805f466aae6ff56-1] regionserver.HStore(310): Store=c44270c3412c22e2c805f466aae6ff56/A, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-15 02:53:34,471 INFO [StoreOpener-c44270c3412c22e2c805f466aae6ff56-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family B of region c44270c3412c22e2c805f466aae6ff56 2023-04-15 02:53:34,473 DEBUG [StoreOpener-c44270c3412c22e2c805f466aae6ff56-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/c44270c3412c22e2c805f466aae6ff56/B 2023-04-15 02:53:34,473 DEBUG [StoreOpener-c44270c3412c22e2c805f466aae6ff56-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/c44270c3412c22e2c805f466aae6ff56/B 2023-04-15 02:53:34,474 INFO [StoreOpener-c44270c3412c22e2c805f466aae6ff56-1] regionserver.CompactingMemStore(121): Store=B, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-15 02:53:34,474 INFO [StoreOpener-c44270c3412c22e2c805f466aae6ff56-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region c44270c3412c22e2c805f466aae6ff56 columnFamilyName B 2023-04-15 02:53:34,475 INFO [StoreOpener-c44270c3412c22e2c805f466aae6ff56-1] regionserver.HStore(310): Store=c44270c3412c22e2c805f466aae6ff56/B, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-15 02:53:34,475 INFO [StoreOpener-c44270c3412c22e2c805f466aae6ff56-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family C of region c44270c3412c22e2c805f466aae6ff56 2023-04-15 02:53:34,477 DEBUG [StoreOpener-c44270c3412c22e2c805f466aae6ff56-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/c44270c3412c22e2c805f466aae6ff56/C 2023-04-15 02:53:34,477 DEBUG [StoreOpener-c44270c3412c22e2c805f466aae6ff56-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/c44270c3412c22e2c805f466aae6ff56/C 2023-04-15 02:53:34,477 INFO [StoreOpener-c44270c3412c22e2c805f466aae6ff56-1] regionserver.CompactingMemStore(121): Store=C, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-15 02:53:34,477 INFO [StoreOpener-c44270c3412c22e2c805f466aae6ff56-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region c44270c3412c22e2c805f466aae6ff56 columnFamilyName C 2023-04-15 02:53:34,478 INFO [StoreOpener-c44270c3412c22e2c805f466aae6ff56-1] regionserver.HStore(310): Store=c44270c3412c22e2c805f466aae6ff56/C, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-15 02:53:34,479 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1146): Setting FlushNonSloppyStoresFirstPolicy for the region=TestAcidGuarantees,,1681527213260.c44270c3412c22e2c805f466aae6ff56. 2023-04-15 02:53:34,481 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/c44270c3412c22e2c805f466aae6ff56 2023-04-15 02:53:34,481 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/c44270c3412c22e2c805f466aae6ff56 2023-04-15 02:53:34,485 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table TestAcidGuarantees descriptor;using region.getMemStoreFlushHeapSize/# of families (16.0 M)) instead. 2023-04-15 02:53:34,488 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1055): writing seq id for c44270c3412c22e2c805f466aae6ff56 2023-04-15 02:53:34,492 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/c44270c3412c22e2c805f466aae6ff56/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-04-15 02:53:34,493 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1072): Opened c44270c3412c22e2c805f466aae6ff56; next sequenceid=2; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=74128904, jitterRate=0.1046067476272583}, FlushLargeStoresPolicy{flushSizeLowerBound=16777216} 2023-04-15 02:53:34,493 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(965): Region open journal for c44270c3412c22e2c805f466aae6ff56: 2023-04-15 02:53:34,495 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for TestAcidGuarantees,,1681527213260.c44270c3412c22e2c805f466aae6ff56., pid=11, masterSystemTime=1681527214451 2023-04-15 02:53:34,497 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for TestAcidGuarantees,,1681527213260.c44270c3412c22e2c805f466aae6ff56. 2023-04-15 02:53:34,497 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] handler.AssignRegionHandler(156): Opened TestAcidGuarantees,,1681527213260.c44270c3412c22e2c805f466aae6ff56. 2023-04-15 02:53:34,498 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=10 updating hbase:meta row=c44270c3412c22e2c805f466aae6ff56, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:34,498 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"TestAcidGuarantees,,1681527213260.c44270c3412c22e2c805f466aae6ff56.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1681527214498"},{"qualifier":"server","vlen":32,"tag":[],"timestamp":"1681527214498"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1681527214498"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1681527214498"}]},"ts":"1681527214498"} 2023-04-15 02:53:34,504 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=11, resume processing ppid=10 2023-04-15 02:53:34,504 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=11, ppid=10, state=SUCCESS; OpenRegionProcedure c44270c3412c22e2c805f466aae6ff56, server=jenkins-hbase20.apache.org,34681,1681527209133 in 204 msec 2023-04-15 02:53:34,507 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=10, resume processing ppid=9 2023-04-15 02:53:34,508 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=10, ppid=9, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=c44270c3412c22e2c805f466aae6ff56, ASSIGN in 366 msec 2023-04-15 02:53:34,509 INFO [PEWorker-4] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_UPDATE_DESC_CACHE, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_UPDATE_DESC_CACHE 2023-04-15 02:53:34,509 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1681527214509"}]},"ts":"1681527214509"} 2023-04-15 02:53:34,511 INFO [PEWorker-4] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=ENABLED in hbase:meta 2023-04-15 02:53:34,514 INFO [PEWorker-4] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_POST_OPERATION, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_POST_OPERATION 2023-04-15 02:53:34,516 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=9, state=SUCCESS; CreateTableProcedure table=TestAcidGuarantees in 1.2500 sec 2023-04-15 02:53:35,410 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(1227): Checking to see if procedure is done pid=9 2023-04-15 02:53:35,411 INFO [Listener at localhost.localdomain/33693] client.HBaseAdmin$TableFuture(3541): Operation: CREATE, Table Name: default:TestAcidGuarantees, procId: 9 completed 2023-04-15 02:53:35,419 DEBUG [Listener at localhost.localdomain/33693] zookeeper.ReadOnlyZKClient(139): Connect 0x7f4385a1 to 127.0.0.1:51762 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-15 02:53:35,423 DEBUG [Listener at localhost.localdomain/33693] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@169fe8bf, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-15 02:53:35,427 DEBUG [Listener at localhost.localdomain/33693] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-15 02:53:35,430 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:38474, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-15 02:53:35,433 DEBUG [Listener at localhost.localdomain/33693] ipc.RpcConnection(124): Using SIMPLE authentication for service=MasterService, sasl=false 2023-04-15 02:53:35,436 INFO [RS-EventLoopGroup-1-1] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:55184, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=MasterService 2023-04-15 02:53:35,444 DEBUG [Listener at localhost.localdomain/33693] zookeeper.ReadOnlyZKClient(139): Connect 0x346ea1a1 to 127.0.0.1:51762 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-15 02:53:35,448 DEBUG [Listener at localhost.localdomain/33693] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@202b48c4, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-15 02:53:35,448 DEBUG [Listener at localhost.localdomain/33693] zookeeper.ReadOnlyZKClient(139): Connect 0x05be6782 to 127.0.0.1:51762 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-15 02:53:35,452 DEBUG [Listener at localhost.localdomain/33693] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@3e0ba002, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-15 02:53:35,453 DEBUG [Listener at localhost.localdomain/33693] zookeeper.ReadOnlyZKClient(139): Connect 0x5610151a to 127.0.0.1:51762 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-15 02:53:35,456 DEBUG [Listener at localhost.localdomain/33693] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@77443fb0, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-15 02:53:35,457 DEBUG [Listener at localhost.localdomain/33693] zookeeper.ReadOnlyZKClient(139): Connect 0x13e837e0 to 127.0.0.1:51762 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-15 02:53:35,461 DEBUG [Listener at localhost.localdomain/33693] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@136956fd, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-15 02:53:35,461 DEBUG [Listener at localhost.localdomain/33693] zookeeper.ReadOnlyZKClient(139): Connect 0x3167991b to 127.0.0.1:51762 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-15 02:53:35,465 DEBUG [Listener at localhost.localdomain/33693] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@4654132d, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-15 02:53:35,467 DEBUG [Listener at localhost.localdomain/33693] zookeeper.ReadOnlyZKClient(139): Connect 0x046b5e40 to 127.0.0.1:51762 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-15 02:53:35,478 DEBUG [Listener at localhost.localdomain/33693] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@29be3c5c, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-15 02:53:35,478 DEBUG [Listener at localhost.localdomain/33693] zookeeper.ReadOnlyZKClient(139): Connect 0x127a87ef to 127.0.0.1:51762 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-15 02:53:35,511 DEBUG [Listener at localhost.localdomain/33693] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@4cbd52be, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-15 02:53:35,512 DEBUG [Listener at localhost.localdomain/33693] zookeeper.ReadOnlyZKClient(139): Connect 0x19e3f14f to 127.0.0.1:51762 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-15 02:53:35,524 DEBUG [Listener at localhost.localdomain/33693] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@3fed686f, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-15 02:53:35,525 DEBUG [Listener at localhost.localdomain/33693] zookeeper.ReadOnlyZKClient(139): Connect 0x38ebb9f9 to 127.0.0.1:51762 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-15 02:53:35,528 DEBUG [Listener at localhost.localdomain/33693] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@3e044e22, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-15 02:53:35,535 DEBUG [hconnection-0x25cd8e7f-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-15 02:53:35,537 DEBUG [hconnection-0x2d45398-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-15 02:53:35,538 DEBUG [hconnection-0x7cbe57fd-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-15 02:53:35,539 DEBUG [hconnection-0x258e48f5-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-15 02:53:35,540 DEBUG [hconnection-0x24a6b8d-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-15 02:53:35,540 DEBUG [hconnection-0x77cf76aa-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-15 02:53:35,541 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:38482, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-15 02:53:35,542 DEBUG [hconnection-0x709f9e55-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-15 02:53:35,547 DEBUG [hconnection-0x4a96275a-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-15 02:53:35,548 DEBUG [hconnection-0x43d8c51a-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-15 02:53:35,548 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:38504, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-15 02:53:35,548 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:38488, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-15 02:53:35,548 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:38486, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-15 02:53:35,549 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:38520, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-15 02:53:35,558 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:38530, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-15 02:53:35,558 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:38526, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-15 02:53:35,559 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:38540, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-15 02:53:35,588 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:38542, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-15 02:53:35,603 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-15 02:53:35,613 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-15 02:53:35,614 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-15 02:53:35,614 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-15 02:53:35,614 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-15 02:53:35,615 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-15 02:53:35,616 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:53:35,616 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:53:35,617 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-15 02:53:35,618 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:35,618 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-15 02:53:35,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-15 02:53:35,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:53:35,618 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:35,618 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-15 02:53:35,618 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:53:35,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-15 02:53:35,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:53:35,619 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-15 02:53:35,622 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-15 02:53:35,627 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-15 02:53:35,627 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-15 02:53:35,629 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-15 02:53:35,630 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-15 02:53:35,630 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-15 02:53:35,633 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681527213260.c44270c3412c22e2c805f466aae6ff56. 2023-04-15 02:53:35,636 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681527213260.c44270c3412c22e2c805f466aae6ff56. started... 2023-04-15 02:53:35,638 INFO [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing c44270c3412c22e2c805f466aae6ff56 3/3 column families, dataSize=33.54 KB heapSize=100.36 KB 2023-04-15 02:53:35,652 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c44270c3412c22e2c805f466aae6ff56, store=A 2023-04-15 02:53:35,663 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-15 02:53:35,664 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c44270c3412c22e2c805f466aae6ff56, store=B 2023-04-15 02:53:35,664 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-15 02:53:35,664 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c44270c3412c22e2c805f466aae6ff56, store=C 2023-04-15 02:53:35,664 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-15 02:53:35,673 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34681] regionserver.HRegion(9158): Flush requested on c44270c3412c22e2c805f466aae6ff56 2023-04-15 02:53:35,673 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681527213260.c44270c3412c22e2c805f466aae6ff56. as already flushing 2023-04-15 02:53:35,781 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c44270c3412c22e2c805f466aae6ff56, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:53:35,782 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34681] ipc.CallRunner(144): callId: 6 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:38520 deadline: 1681527275737, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c44270c3412c22e2c805f466aae6ff56, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:35,785 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c44270c3412c22e2c805f466aae6ff56, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:53:35,786 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34681] ipc.CallRunner(144): callId: 6 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:38504 deadline: 1681527275762, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c44270c3412c22e2c805f466aae6ff56, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:35,787 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c44270c3412c22e2c805f466aae6ff56, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:53:35,788 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34681] ipc.CallRunner(144): callId: 6 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:38542 deadline: 1681527275782, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c44270c3412c22e2c805f466aae6ff56, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:35,788 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c44270c3412c22e2c805f466aae6ff56, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:53:35,789 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34681] ipc.CallRunner(144): callId: 6 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:38530 deadline: 1681527275782, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c44270c3412c22e2c805f466aae6ff56, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:35,789 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c44270c3412c22e2c805f466aae6ff56, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:53:35,789 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34681] ipc.CallRunner(144): callId: 7 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:38540 deadline: 1681527275782, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c44270c3412c22e2c805f466aae6ff56, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:35,793 INFO [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=13.42 KB at sequenceid=10 (bloomFilter=true), to=hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/c44270c3412c22e2c805f466aae6ff56/.tmp/A/acacce0fff2a4181b0d8c913048e49e0 2023-04-15 02:53:35,793 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=150, cellSize=6870, totalHeapSize=15840, min timestamp=1681527215721, max timestamp=1681527215760Num uniques 859; 2023-04-15 02:53:35,795 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-1] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=6870, suffix heap size=117064,new segment heap size=15840 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=150 2023-04-15 02:53:35,797 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=150, cellSize=6870, totalHeapSize=15840, min timestamp=1681527215721, max timestamp=1681527215765Num uniques 859; 2023-04-15 02:53:35,797 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=150, cellSize=6870, totalHeapSize=15840, min timestamp=1681527215721, max timestamp=1681527215761Num uniques 859; 2023-04-15 02:53:35,797 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-2] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=6870, suffix heap size=117064,new segment heap size=15840 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=150 2023-04-15 02:53:35,797 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=6870, suffix heap size=117064,new segment heap size=15840 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=150 2023-04-15 02:53:35,902 INFO [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=13.42 KB at sequenceid=10 (bloomFilter=true), to=hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/c44270c3412c22e2c805f466aae6ff56/.tmp/B/e9337a469b0144028bebf63d83f2cafb 2023-04-15 02:53:35,950 INFO [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=13.42 KB at sequenceid=10 (bloomFilter=true), to=hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/c44270c3412c22e2c805f466aae6ff56/.tmp/C/081cf4a0d52047808a158eecff6304fe 2023-04-15 02:53:35,966 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/c44270c3412c22e2c805f466aae6ff56/.tmp/A/acacce0fff2a4181b0d8c913048e49e0 as hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/c44270c3412c22e2c805f466aae6ff56/A/acacce0fff2a4181b0d8c913048e49e0 2023-04-15 02:53:35,982 INFO [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/c44270c3412c22e2c805f466aae6ff56/A/acacce0fff2a4181b0d8c913048e49e0, entries=150, sequenceid=10, filesize=11.6 K 2023-04-15 02:53:35,989 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=168, cellSize=7688, totalHeapSize=17712, min timestamp=1681527215969, max timestamp=1681527215979Num uniques 1009; 2023-04-15 02:53:35,995 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-3] regionserver.CompactionPipeline(163): Suffix data size=46213, new segment data size=7688, suffix heap size=132904,new segment heap size=17712 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1009, new segment cells count=168 2023-04-15 02:53:35,997 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=150, cellSize=6870, totalHeapSize=15840, min timestamp=1681527215969, max timestamp=1681527215979Num uniques 1009; 2023-04-15 02:53:35,997 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-4] regionserver.CompactionPipeline(163): Suffix data size=46213, new segment data size=6870, suffix heap size=132904,new segment heap size=15840 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1009, new segment cells count=150 2023-04-15 02:53:35,999 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=150, cellSize=6870, totalHeapSize=15840, min timestamp=1681527215969, max timestamp=1681527215979Num uniques 1009; 2023-04-15 02:53:35,999 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-5] regionserver.CompactionPipeline(163): Suffix data size=46213, new segment data size=6870, suffix heap size=132904,new segment heap size=15840 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1009, new segment cells count=150 2023-04-15 02:53:36,001 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/c44270c3412c22e2c805f466aae6ff56/.tmp/B/e9337a469b0144028bebf63d83f2cafb as hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/c44270c3412c22e2c805f466aae6ff56/B/e9337a469b0144028bebf63d83f2cafb 2023-04-15 02:53:36,019 INFO [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/c44270c3412c22e2c805f466aae6ff56/B/e9337a469b0144028bebf63d83f2cafb, entries=150, sequenceid=10, filesize=11.6 K 2023-04-15 02:53:36,024 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/c44270c3412c22e2c805f466aae6ff56/.tmp/C/081cf4a0d52047808a158eecff6304fe as hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/c44270c3412c22e2c805f466aae6ff56/C/081cf4a0d52047808a158eecff6304fe 2023-04-15 02:53:36,038 INFO [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/c44270c3412c22e2c805f466aae6ff56/C/081cf4a0d52047808a158eecff6304fe, entries=150, sequenceid=10, filesize=11.6 K 2023-04-15 02:53:36,042 INFO [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~40.25 KB/41220, heapSize ~120.23 KB/123120, currentSize=58.78 KB/60194 for c44270c3412c22e2c805f466aae6ff56 in 403ms, sequenceid=10, compaction requested=false 2023-04-15 02:53:36,043 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.MetricsTableSourceImpl(130): Creating new MetricsTableSourceImpl for table 'TestAcidGuarantees' 2023-04-15 02:53:36,046 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for c44270c3412c22e2c805f466aae6ff56: 2023-04-15 02:53:36,046 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681527213260.c44270c3412c22e2c805f466aae6ff56. 2023-04-15 02:53:36,046 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-15 02:53:36,047 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-15 02:53:36,047 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:36,047 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34681] regionserver.HRegion(9158): Flush requested on c44270c3412c22e2c805f466aae6ff56 2023-04-15 02:53:36,048 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-15 02:53:36,048 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34681,1681527209133' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-15 02:53:36,051 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing c44270c3412c22e2c805f466aae6ff56 3/3 column families, dataSize=65.49 KB heapSize=181.27 KB 2023-04-15 02:53:36,052 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c44270c3412c22e2c805f466aae6ff56, store=A 2023-04-15 02:53:36,052 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-15 02:53:36,054 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:53:36,054 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:36,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:36,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:53:36,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:53:36,055 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:53:36,055 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-15 02:53:36,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:53:36,054 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c44270c3412c22e2c805f466aae6ff56, store=B 2023-04-15 02:53:36,056 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-15 02:53:36,056 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c44270c3412c22e2c805f466aae6ff56, store=C 2023-04-15 02:53:36,056 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-15 02:53:36,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:53:36,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:53:36,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:36,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:53:36,063 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34681,1681527209133' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-15 02:53:36,064 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-15 02:53:36,064 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:53:36,064 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5ec5e294[Count = 0] remaining members to acquire global barrier 2023-04-15 02:53:36,072 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:53:36,072 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:53:36,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:53:36,072 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:36,072 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-15 02:53:36,072 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-15 02:53:36,073 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-15 02:53:36,073 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34681,1681527209133' in zk 2023-04-15 02:53:36,079 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:36,079 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-15 02:53:36,079 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-15 02:53:36,079 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-15 02:53:36,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:36,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:53:36,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:53:36,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:53:36,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:53:36,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:53:36,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:36,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:53:36,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:53:36,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:36,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34681,1681527209133': 2023-04-15 02:53:36,091 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34681,1681527209133' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-15 02:53:36,091 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-15 02:53:36,091 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-15 02:53:36,091 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-15 02:53:36,091 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-15 02:53:36,091 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-15 02:53:36,113 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:53:36,113 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:53:36,113 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:53:36,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:53:36,113 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-15 02:53:36,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:53:36,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:53:36,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:53:36,114 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-15 02:53:36,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-15 02:53:36,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:53:36,117 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:36,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:53:36,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:53:36,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:53:36,131 DEBUG [HBase-Metrics2-1] regionserver.MetricsTableSourceImpl(130): Creating new MetricsTableSourceImpl for table 'hbase:meta' 2023-04-15 02:53:36,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:53:36,132 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:36,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:36,132 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=54 (bloomFilter=true), to=hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/c44270c3412c22e2c805f466aae6ff56/.tmp/A/6dd7dd3e6325495f90a18096bd9a44a8 2023-04-15 02:53:36,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:53:36,133 DEBUG [HBase-Metrics2-1] regionserver.MetricsTableSourceImpl(130): Creating new MetricsTableSourceImpl for table 'hbase:namespace' 2023-04-15 02:53:36,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:53:36,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:36,155 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:36,156 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:53:36,156 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:53:36,156 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:36,157 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:53:36,157 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:53:36,157 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:53:36,155 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-15 02:53:36,157 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-15 02:53:36,155 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-15 02:53:36,158 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-15 02:53:36,158 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-15 02:53:36,155 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-15 02:53:36,158 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-15 02:53:36,159 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-15 02:53:36,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-15 02:53:36,159 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-15 02:53:36,160 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-15 02:53:36,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-15 02:53:36,170 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c44270c3412c22e2c805f466aae6ff56, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:53:36,171 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34681] ipc.CallRunner(144): callId: 15 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:38530 deadline: 1681527276158, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c44270c3412c22e2c805f466aae6ff56, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:36,200 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c44270c3412c22e2c805f466aae6ff56, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:53:36,206 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34681] ipc.CallRunner(144): callId: 15 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:38542 deadline: 1681527276160, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c44270c3412c22e2c805f466aae6ff56, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:36,206 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=21.56 KB at sequenceid=54 (bloomFilter=true), to=hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/c44270c3412c22e2c805f466aae6ff56/.tmp/B/fe072ba56cc048e29f7a020e4e4f8eea 2023-04-15 02:53:36,207 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c44270c3412c22e2c805f466aae6ff56, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:53:36,207 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34681] ipc.CallRunner(144): callId: 16 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:38520 deadline: 1681527276162, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c44270c3412c22e2c805f466aae6ff56, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:36,208 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c44270c3412c22e2c805f466aae6ff56, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:53:36,209 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34681] ipc.CallRunner(144): callId: 17 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:38540 deadline: 1681527276165, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c44270c3412c22e2c805f466aae6ff56, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:36,209 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c44270c3412c22e2c805f466aae6ff56, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:53:36,210 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34681] ipc.CallRunner(144): callId: 16 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:38504 deadline: 1681527276171, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c44270c3412c22e2c805f466aae6ff56, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:36,259 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-15 02:53:36,267 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-15 02:53:36,280 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-15 02:53:36,288 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c44270c3412c22e2c805f466aae6ff56, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:53:36,289 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34681] ipc.CallRunner(144): callId: 17 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:38530 deadline: 1681527276276, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c44270c3412c22e2c805f466aae6ff56, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:36,296 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-15 02:53:36,296 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-15 02:53:36,296 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-15 02:53:36,297 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-15 02:53:36,298 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-15 02:53:36,298 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-15 02:53:36,299 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:53:36,299 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:53:36,300 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-15 02:53:36,300 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-15 02:53:36,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-15 02:53:36,301 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:36,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:53:36,302 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:53:36,302 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:36,302 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-15 02:53:36,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-15 02:53:36,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:53:36,303 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-15 02:53:36,303 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-15 02:53:36,303 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-15 02:53:36,303 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-15 02:53:36,303 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-15 02:53:36,306 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-15 02:53:36,307 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-15 02:53:36,307 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-15 02:53:36,307 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681527213260.c44270c3412c22e2c805f466aae6ff56. 2023-04-15 02:53:36,307 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681527213260.c44270c3412c22e2c805f466aae6ff56. started... 2023-04-15 02:53:36,307 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681527213260.c44270c3412c22e2c805f466aae6ff56. as already flushing 2023-04-15 02:53:36,307 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681527213260.c44270c3412c22e2c805f466aae6ff56. 2023-04-15 02:53:36,308 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-15 02:53:36,308 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-15 02:53:36,308 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:36,308 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-15 02:53:36,308 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34681,1681527209133' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-15 02:53:36,311 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:36,311 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:53:36,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:36,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:53:36,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:53:36,312 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:53:36,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:53:36,312 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-15 02:53:36,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:53:36,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:53:36,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:36,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:53:36,315 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34681,1681527209133' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-15 02:53:36,315 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@463e1032[Count = 0] remaining members to acquire global barrier 2023-04-15 02:53:36,315 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-15 02:53:36,315 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:53:36,321 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c44270c3412c22e2c805f466aae6ff56, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:53:36,321 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34681] ipc.CallRunner(144): callId: 18 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:38504 deadline: 1681527276313, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c44270c3412c22e2c805f466aae6ff56, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:36,323 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:53:36,324 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:53:36,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:53:36,324 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:36,324 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-15 02:53:36,324 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-15 02:53:36,324 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-15 02:53:36,324 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34681,1681527209133' in zk 2023-04-15 02:53:36,327 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c44270c3412c22e2c805f466aae6ff56, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:53:36,331 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34681] ipc.CallRunner(144): callId: 17 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:38542 deadline: 1681527276315, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c44270c3412c22e2c805f466aae6ff56, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:36,331 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-15 02:53:36,331 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-15 02:53:36,331 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-15 02:53:36,331 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:36,331 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c44270c3412c22e2c805f466aae6ff56, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:53:36,331 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c44270c3412c22e2c805f466aae6ff56, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:53:36,332 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34681] ipc.CallRunner(144): callId: 19 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:38540 deadline: 1681527276320, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c44270c3412c22e2c805f466aae6ff56, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:36,332 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34681] ipc.CallRunner(144): callId: 18 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:38520 deadline: 1681527276318, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c44270c3412c22e2c805f466aae6ff56, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:36,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:36,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:53:36,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:53:36,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:53:36,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:53:36,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:53:36,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:36,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:53:36,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:53:36,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:36,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34681,1681527209133': 2023-04-15 02:53:36,341 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34681,1681527209133' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-15 02:53:36,341 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-15 02:53:36,341 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-15 02:53:36,341 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-15 02:53:36,341 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-15 02:53:36,341 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-15 02:53:36,344 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:53:36,344 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-15 02:53:36,344 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:53:36,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:53:36,345 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-15 02:53:36,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-15 02:53:36,345 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:53:36,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:53:36,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:53:36,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:53:36,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:53:36,346 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:36,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:53:36,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:53:36,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:53:36,349 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:36,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:53:36,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:36,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:53:36,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:53:36,357 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-15 02:53:36,357 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-15 02:53:36,357 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-15 02:53:36,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-15 02:53:36,358 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:36,358 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:53:36,358 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:53:36,358 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:36,358 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:53:36,359 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:53:36,359 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:53:36,359 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-15 02:53:36,359 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-15 02:53:36,359 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-15 02:53:36,359 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-15 02:53:36,359 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-15 02:53:36,358 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-15 02:53:36,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-15 02:53:36,360 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-15 02:53:36,460 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-15 02:53:36,461 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-15 02:53:36,466 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-15 02:53:36,478 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-15 02:53:36,478 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-15 02:53:36,478 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-15 02:53:36,478 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-15 02:53:36,479 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-15 02:53:36,479 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-15 02:53:36,480 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:53:36,480 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:53:36,481 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-15 02:53:36,481 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:36,481 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-15 02:53:36,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-15 02:53:36,482 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:36,482 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-15 02:53:36,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:53:36,482 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:53:36,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-15 02:53:36,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:53:36,483 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-15 02:53:36,486 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-15 02:53:36,486 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-15 02:53:36,486 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-15 02:53:36,486 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-15 02:53:36,490 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-15 02:53:36,491 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-15 02:53:36,491 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-15 02:53:36,491 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681527213260.c44270c3412c22e2c805f466aae6ff56. 2023-04-15 02:53:36,496 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c44270c3412c22e2c805f466aae6ff56, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:53:36,496 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681527213260.c44270c3412c22e2c805f466aae6ff56. started... 2023-04-15 02:53:36,497 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34681] ipc.CallRunner(144): callId: 19 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:38530 deadline: 1681527276495, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c44270c3412c22e2c805f466aae6ff56, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:36,497 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681527213260.c44270c3412c22e2c805f466aae6ff56. as already flushing 2023-04-15 02:53:36,497 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681527213260.c44270c3412c22e2c805f466aae6ff56. 2023-04-15 02:53:36,498 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-15 02:53:36,498 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-15 02:53:36,498 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:36,498 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-15 02:53:36,498 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34681,1681527209133' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-15 02:53:36,503 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:36,503 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:53:36,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:36,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:53:36,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:53:36,504 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:53:36,505 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-15 02:53:36,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:53:36,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:53:36,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:53:36,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:36,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:53:36,509 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34681,1681527209133' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-15 02:53:36,509 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3c335f27[Count = 0] remaining members to acquire global barrier 2023-04-15 02:53:36,509 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-15 02:53:36,509 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:53:36,511 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:53:36,511 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:53:36,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:53:36,511 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-15 02:53:36,511 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-15 02:53:36,511 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34681,1681527209133' in zk 2023-04-15 02:53:36,511 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:36,511 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-15 02:53:36,514 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-15 02:53:36,514 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-15 02:53:36,514 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-15 02:53:36,518 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:36,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:36,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:53:36,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:53:36,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:53:36,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:53:36,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:53:36,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:36,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:53:36,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:53:36,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:36,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34681,1681527209133': 2023-04-15 02:53:36,524 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34681,1681527209133' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-15 02:53:36,524 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-15 02:53:36,524 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-15 02:53:36,524 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-15 02:53:36,524 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-15 02:53:36,524 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-15 02:53:36,527 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c44270c3412c22e2c805f466aae6ff56, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:53:36,528 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:53:36,528 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-15 02:53:36,528 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:53:36,528 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34681] ipc.CallRunner(144): callId: 20 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:38504 deadline: 1681527276525, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c44270c3412c22e2c805f466aae6ff56, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:36,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:53:36,529 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:53:36,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:53:36,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:53:36,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:53:36,530 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-15 02:53:36,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:53:36,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-15 02:53:36,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:53:36,531 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:36,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:53:36,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:53:36,537 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c44270c3412c22e2c805f466aae6ff56, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:53:36,537 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34681] ipc.CallRunner(144): callId: 19 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:38542 deadline: 1681527276535, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c44270c3412c22e2c805f466aae6ff56, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:36,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:53:36,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:36,539 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:36,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:53:36,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:53:36,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:36,548 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c44270c3412c22e2c805f466aae6ff56, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:53:36,549 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34681] ipc.CallRunner(144): callId: 20 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:38520 deadline: 1681527276537, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c44270c3412c22e2c805f466aae6ff56, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:36,549 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c44270c3412c22e2c805f466aae6ff56, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:53:36,554 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34681] ipc.CallRunner(144): callId: 21 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:38540 deadline: 1681527276537, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c44270c3412c22e2c805f466aae6ff56, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:36,555 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:36,555 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-15 02:53:36,555 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-15 02:53:36,555 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-15 02:53:36,555 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-15 02:53:36,555 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-15 02:53:36,555 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-15 02:53:36,555 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-15 02:53:36,555 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:53:36,556 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:53:36,556 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:36,556 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:53:36,556 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:53:36,556 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-15 02:53:36,555 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-15 02:53:36,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-15 02:53:36,556 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:53:36,557 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-15 02:53:36,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-15 02:53:36,656 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-15 02:53:36,657 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-15 02:53:36,662 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-15 02:53:36,672 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=21.56 KB at sequenceid=54 (bloomFilter=true), to=hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/c44270c3412c22e2c805f466aae6ff56/.tmp/C/767f544ff04444b4b1e74da69ea1dc48 2023-04-15 02:53:36,678 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-15 02:53:36,678 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-15 02:53:36,678 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-15 02:53:36,678 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-15 02:53:36,679 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-15 02:53:36,679 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-15 02:53:36,680 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:53:36,680 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:53:36,681 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-15 02:53:36,681 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:36,681 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-15 02:53:36,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-15 02:53:36,682 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:36,682 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-15 02:53:36,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:53:36,682 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:53:36,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-15 02:53:36,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:53:36,683 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-15 02:53:36,683 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-15 02:53:36,683 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-15 02:53:36,683 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-15 02:53:36,684 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-15 02:53:36,687 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-15 02:53:36,688 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-15 02:53:36,688 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-15 02:53:36,688 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681527213260.c44270c3412c22e2c805f466aae6ff56. 2023-04-15 02:53:36,688 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681527213260.c44270c3412c22e2c805f466aae6ff56. started... 2023-04-15 02:53:36,688 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681527213260.c44270c3412c22e2c805f466aae6ff56. as already flushing 2023-04-15 02:53:36,688 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681527213260.c44270c3412c22e2c805f466aae6ff56. 2023-04-15 02:53:36,689 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-15 02:53:36,689 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-15 02:53:36,689 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:36,689 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-15 02:53:36,689 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34681,1681527209133' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-15 02:53:36,693 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/c44270c3412c22e2c805f466aae6ff56/.tmp/A/6dd7dd3e6325495f90a18096bd9a44a8 as hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/c44270c3412c22e2c805f466aae6ff56/A/6dd7dd3e6325495f90a18096bd9a44a8 2023-04-15 02:53:36,694 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:36,694 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:53:36,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:36,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:53:36,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:53:36,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:53:36,695 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:53:36,695 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-15 02:53:36,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:53:36,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:53:36,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:36,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:53:36,697 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34681,1681527209133' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-15 02:53:36,697 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2f9e6cbd[Count = 0] remaining members to acquire global barrier 2023-04-15 02:53:36,697 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-15 02:53:36,697 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:53:36,698 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:53:36,698 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:53:36,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:53:36,698 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-15 02:53:36,698 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-15 02:53:36,698 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34681,1681527209133' in zk 2023-04-15 02:53:36,698 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:36,699 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-15 02:53:36,700 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:36,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:36,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:53:36,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:53:36,701 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-15 02:53:36,701 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-15 02:53:36,701 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-15 02:53:36,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:53:36,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:53:36,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:53:36,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:36,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:53:36,704 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/c44270c3412c22e2c805f466aae6ff56/A/6dd7dd3e6325495f90a18096bd9a44a8, entries=250, sequenceid=54, filesize=16.2 K 2023-04-15 02:53:36,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:53:36,706 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/c44270c3412c22e2c805f466aae6ff56/.tmp/B/fe072ba56cc048e29f7a020e4e4f8eea as hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/c44270c3412c22e2c805f466aae6ff56/B/fe072ba56cc048e29f7a020e4e4f8eea 2023-04-15 02:53:36,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:36,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34681,1681527209133': 2023-04-15 02:53:36,708 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34681,1681527209133' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-15 02:53:36,708 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-15 02:53:36,709 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-15 02:53:36,709 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-15 02:53:36,709 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-15 02:53:36,709 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-15 02:53:36,710 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:53:36,711 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:53:36,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:53:36,711 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:53:36,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:53:36,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:53:36,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:53:36,711 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-15 02:53:36,711 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-15 02:53:36,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-15 02:53:36,711 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:36,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:53:36,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:53:36,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:53:36,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:53:36,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:53:36,720 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:36,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:36,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:53:36,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:53:36,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:36,731 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-15 02:53:36,732 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-15 02:53:36,732 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-15 02:53:36,733 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-15 02:53:36,733 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-15 02:53:36,734 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-15 02:53:36,734 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-15 02:53:36,734 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-15 02:53:36,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-15 02:53:36,734 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:36,734 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:53:36,734 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:53:36,734 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:36,734 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:53:36,734 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:53:36,734 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:53:36,735 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-15 02:53:36,735 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/c44270c3412c22e2c805f466aae6ff56/B/fe072ba56cc048e29f7a020e4e4f8eea, entries=150, sequenceid=54, filesize=11.6 K 2023-04-15 02:53:36,739 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/c44270c3412c22e2c805f466aae6ff56/.tmp/C/767f544ff04444b4b1e74da69ea1dc48 as hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/c44270c3412c22e2c805f466aae6ff56/C/767f544ff04444b4b1e74da69ea1dc48 2023-04-15 02:53:36,743 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-15 02:53:36,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-15 02:53:36,749 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/c44270c3412c22e2c805f466aae6ff56/C/767f544ff04444b4b1e74da69ea1dc48, entries=150, sequenceid=54, filesize=11.6 K 2023-04-15 02:53:36,751 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~65.49 KB/67064, heapSize ~181.22 KB/185568, currentSize=114.05 KB/116790 for c44270c3412c22e2c805f466aae6ff56 in 703ms, sequenceid=54, compaction requested=false 2023-04-15 02:53:36,759 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for c44270c3412c22e2c805f466aae6ff56: 2023-04-15 02:53:36,817 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681527216147, max timestamp=1681527216805Num uniques 859; 2023-04-15 02:53:36,817 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-7] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-15 02:53:36,817 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681527216147, max timestamp=1681527216805Num uniques 859; 2023-04-15 02:53:36,818 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-6] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-15 02:53:36,818 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34681] regionserver.HRegion(9158): Flush requested on c44270c3412c22e2c805f466aae6ff56 2023-04-15 02:53:36,818 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing c44270c3412c22e2c805f466aae6ff56 3/3 column families, dataSize=58.14 KB heapSize=164.17 KB 2023-04-15 02:53:36,818 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c44270c3412c22e2c805f466aae6ff56, store=A 2023-04-15 02:53:36,818 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-15 02:53:36,818 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c44270c3412c22e2c805f466aae6ff56, store=B 2023-04-15 02:53:36,819 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-15 02:53:36,819 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c44270c3412c22e2c805f466aae6ff56, store=C 2023-04-15 02:53:36,819 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-15 02:53:36,833 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-15 02:53:36,834 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-15 02:53:36,838 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-15 02:53:36,849 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-15 02:53:36,849 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-15 02:53:36,849 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-15 02:53:36,849 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-15 02:53:36,850 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-15 02:53:36,850 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-15 02:53:36,851 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:53:36,851 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:53:36,852 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:36,852 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-15 02:53:36,852 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-15 02:53:36,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-15 02:53:36,854 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:36,854 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-15 02:53:36,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:53:36,857 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:53:36,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-15 02:53:36,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:53:36,860 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-15 02:53:36,865 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-15 02:53:36,865 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-15 02:53:36,865 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-15 02:53:36,865 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-15 02:53:36,867 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-15 02:53:36,870 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-15 02:53:36,870 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-15 02:53:36,873 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681527213260.c44270c3412c22e2c805f466aae6ff56. 2023-04-15 02:53:36,873 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681527213260.c44270c3412c22e2c805f466aae6ff56. started... 2023-04-15 02:53:36,873 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681527213260.c44270c3412c22e2c805f466aae6ff56. as already flushing 2023-04-15 02:53:36,874 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681527213260.c44270c3412c22e2c805f466aae6ff56. 2023-04-15 02:53:36,875 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-15 02:53:36,875 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-15 02:53:36,875 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:36,875 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-15 02:53:36,875 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34681,1681527209133' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-15 02:53:36,880 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=75 (bloomFilter=true), to=hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/c44270c3412c22e2c805f466aae6ff56/.tmp/A/3bbb13c5de744a0fa342f72bc2c1221f 2023-04-15 02:53:36,880 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:36,880 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:53:36,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:36,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:53:36,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:53:36,881 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:53:36,881 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-15 02:53:36,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:53:36,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:53:36,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:53:36,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:36,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:53:36,883 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34681,1681527209133' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-15 02:53:36,883 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5e3efa98[Count = 0] remaining members to acquire global barrier 2023-04-15 02:53:36,883 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-15 02:53:36,883 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:53:36,885 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:53:36,885 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:53:36,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:53:36,885 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-15 02:53:36,885 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-15 02:53:36,885 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34681,1681527209133' in zk 2023-04-15 02:53:36,887 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:36,887 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-15 02:53:36,887 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:36,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:36,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:53:36,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:53:36,887 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-15 02:53:36,888 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-15 02:53:36,888 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-15 02:53:36,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:53:36,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:53:36,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:53:36,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:36,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:53:36,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:53:36,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:36,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34681,1681527209133': 2023-04-15 02:53:36,893 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34681,1681527209133' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-15 02:53:36,893 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-15 02:53:36,893 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-15 02:53:36,893 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-15 02:53:36,893 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-15 02:53:36,893 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-15 02:53:36,899 WARN [HBase-Metrics2-1] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties 2023-04-15 02:53:36,902 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:53:36,902 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:53:36,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:53:36,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:53:36,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:53:36,903 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:53:36,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:53:36,903 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-15 02:53:36,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:53:36,905 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-15 02:53:36,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-15 02:53:36,906 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:36,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:53:36,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:53:36,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:53:36,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:53:36,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:36,910 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:36,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:53:36,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:53:36,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:36,914 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:36,917 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:53:36,917 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-15 02:53:36,917 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-15 02:53:36,917 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-15 02:53:36,917 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:53:36,917 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-15 02:53:36,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-15 02:53:36,917 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-15 02:53:36,917 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:36,918 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:53:36,918 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:53:36,918 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:53:36,917 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-15 02:53:36,918 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-15 02:53:36,918 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-15 02:53:36,919 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-15 02:53:36,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-15 02:53:36,947 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681527216922, max timestamp=1681527216944Num uniques 859; 2023-04-15 02:53:36,947 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681527216922, max timestamp=1681527216944Num uniques 859; 2023-04-15 02:53:36,947 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-1] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-15 02:53:36,947 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-2] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-15 02:53:36,947 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=75 (bloomFilter=true), to=hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/c44270c3412c22e2c805f466aae6ff56/.tmp/B/b9105532a32d4897a770fbf662f69001 2023-04-15 02:53:36,975 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=150, cellSize=6870, totalHeapSize=15840, min timestamp=1681527216922, max timestamp=1681527216940Num uniques 859; 2023-04-15 02:53:36,976 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-9] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=6870, suffix heap size=117064,new segment heap size=15840 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=150 2023-04-15 02:53:36,996 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c44270c3412c22e2c805f466aae6ff56, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:53:36,996 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34681] ipc.CallRunner(144): callId: 30 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:38530 deadline: 1681527276983, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c44270c3412c22e2c805f466aae6ff56, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:36,997 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c44270c3412c22e2c805f466aae6ff56, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:53:36,997 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34681] ipc.CallRunner(144): callId: 28 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:38520 deadline: 1681527276984, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c44270c3412c22e2c805f466aae6ff56, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:36,997 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c44270c3412c22e2c805f466aae6ff56, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:53:36,997 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34681] ipc.CallRunner(144): callId: 27 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:38542 deadline: 1681527276989, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c44270c3412c22e2c805f466aae6ff56, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:37,008 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c44270c3412c22e2c805f466aae6ff56, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:53:37,008 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34681] ipc.CallRunner(144): callId: 29 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:38504 deadline: 1681527276991, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c44270c3412c22e2c805f466aae6ff56, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:37,012 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c44270c3412c22e2c805f466aae6ff56, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:53:37,012 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34681] ipc.CallRunner(144): callId: 29 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:38540 deadline: 1681527276998, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c44270c3412c22e2c805f466aae6ff56, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:37,018 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-15 02:53:37,020 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-15 02:53:37,030 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-15 02:53:37,035 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-15 02:53:37,035 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-15 02:53:37,035 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-15 02:53:37,035 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=40.25 KB at sequenceid=75 (bloomFilter=true), to=hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/c44270c3412c22e2c805f466aae6ff56/.tmp/C/0789443aa76c4c5a845f56519c2f3c41 2023-04-15 02:53:37,035 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-15 02:53:37,036 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-15 02:53:37,036 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-15 02:53:37,037 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:53:37,037 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:53:37,038 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:37,038 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-15 02:53:37,038 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-15 02:53:37,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-15 02:53:37,039 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:37,039 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-15 02:53:37,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:53:37,039 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:53:37,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-15 02:53:37,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:53:37,040 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-15 02:53:37,040 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-15 02:53:37,040 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-15 02:53:37,040 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-15 02:53:37,040 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-15 02:53:37,041 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-15 02:53:37,041 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-15 02:53:37,041 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-15 02:53:37,041 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681527213260.c44270c3412c22e2c805f466aae6ff56. 2023-04-15 02:53:37,041 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681527213260.c44270c3412c22e2c805f466aae6ff56. started... 2023-04-15 02:53:37,042 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681527213260.c44270c3412c22e2c805f466aae6ff56. as already flushing 2023-04-15 02:53:37,042 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681527213260.c44270c3412c22e2c805f466aae6ff56. 2023-04-15 02:53:37,042 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-15 02:53:37,042 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-15 02:53:37,042 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:37,043 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-15 02:53:37,043 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34681,1681527209133' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-15 02:53:37,057 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:37,057 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:53:37,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:37,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:53:37,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:53:37,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:53:37,059 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:53:37,059 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-15 02:53:37,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:53:37,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:53:37,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:37,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:53:37,061 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/c44270c3412c22e2c805f466aae6ff56/.tmp/A/3bbb13c5de744a0fa342f72bc2c1221f as hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/c44270c3412c22e2c805f466aae6ff56/A/3bbb13c5de744a0fa342f72bc2c1221f 2023-04-15 02:53:37,063 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34681,1681527209133' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-15 02:53:37,063 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@68b7ceed[Count = 0] remaining members to acquire global barrier 2023-04-15 02:53:37,063 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-15 02:53:37,063 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:53:37,065 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:53:37,065 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:53:37,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:53:37,066 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:37,066 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-15 02:53:37,066 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-15 02:53:37,070 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-15 02:53:37,071 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34681,1681527209133' in zk 2023-04-15 02:53:37,077 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:37,077 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-15 02:53:37,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:37,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:53:37,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:53:37,077 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-15 02:53:37,078 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-15 02:53:37,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:53:37,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:53:37,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:53:37,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:37,080 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/c44270c3412c22e2c805f466aae6ff56/A/3bbb13c5de744a0fa342f72bc2c1221f, entries=150, sequenceid=75, filesize=11.6 K 2023-04-15 02:53:37,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:53:37,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:53:37,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:37,082 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/c44270c3412c22e2c805f466aae6ff56/.tmp/B/b9105532a32d4897a770fbf662f69001 as hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/c44270c3412c22e2c805f466aae6ff56/B/b9105532a32d4897a770fbf662f69001 2023-04-15 02:53:37,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34681,1681527209133': 2023-04-15 02:53:37,091 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34681,1681527209133' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-15 02:53:37,091 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-15 02:53:37,092 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-15 02:53:37,092 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-15 02:53:37,092 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-15 02:53:37,092 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-15 02:53:37,094 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:53:37,094 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:53:37,094 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:53:37,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:53:37,094 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-15 02:53:37,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:53:37,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:53:37,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:53:37,096 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:37,096 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-15 02:53:37,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-15 02:53:37,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:53:37,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:53:37,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:53:37,098 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:37,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:53:37,102 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:37,102 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:53:37,102 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:53:37,102 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-15 02:53:37,102 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:37,103 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-15 02:53:37,103 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-15 02:53:37,103 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-15 02:53:37,102 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-15 02:53:37,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:53:37,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-15 02:53:37,103 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-15 02:53:37,103 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:53:37,103 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-15 02:53:37,103 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:53:37,106 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-15 02:53:37,107 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:53:37,108 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-15 02:53:37,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-15 02:53:37,109 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-15 02:53:37,109 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c44270c3412c22e2c805f466aae6ff56, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:53:37,109 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34681] ipc.CallRunner(144): callId: 32 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:38530 deadline: 1681527277099, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c44270c3412c22e2c805f466aae6ff56, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:37,110 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c44270c3412c22e2c805f466aae6ff56, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:53:37,111 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34681] ipc.CallRunner(144): callId: 30 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:38520 deadline: 1681527277100, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c44270c3412c22e2c805f466aae6ff56, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:37,111 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c44270c3412c22e2c805f466aae6ff56, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:53:37,112 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34681] ipc.CallRunner(144): callId: 29 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:38542 deadline: 1681527277101, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c44270c3412c22e2c805f466aae6ff56, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:37,115 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c44270c3412c22e2c805f466aae6ff56, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:53:37,115 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34681] ipc.CallRunner(144): callId: 31 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:38504 deadline: 1681527277113, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c44270c3412c22e2c805f466aae6ff56, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:37,116 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c44270c3412c22e2c805f466aae6ff56, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:53:37,117 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/c44270c3412c22e2c805f466aae6ff56/B/b9105532a32d4897a770fbf662f69001, entries=150, sequenceid=75, filesize=11.6 K 2023-04-15 02:53:37,118 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34681] ipc.CallRunner(144): callId: 31 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:38540 deadline: 1681527277115, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c44270c3412c22e2c805f466aae6ff56, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:37,120 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/c44270c3412c22e2c805f466aae6ff56/.tmp/C/0789443aa76c4c5a845f56519c2f3c41 as hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/c44270c3412c22e2c805f466aae6ff56/C/0789443aa76c4c5a845f56519c2f3c41 2023-04-15 02:53:37,137 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/c44270c3412c22e2c805f466aae6ff56/C/0789443aa76c4c5a845f56519c2f3c41, entries=150, sequenceid=75, filesize=11.6 K 2023-04-15 02:53:37,140 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~58.14 KB/59540, heapSize ~164.13 KB/168064, currentSize=127.08 KB/130125 for c44270c3412c22e2c805f466aae6ff56 in 321ms, sequenceid=75, compaction requested=true 2023-04-15 02:53:37,140 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for c44270c3412c22e2c805f466aae6ff56: 2023-04-15 02:53:37,144 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-15 02:53:37,144 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-15 02:53:37,144 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-15 02:53:37,145 DEBUG [RS:0;jenkins-hbase20:34681-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-15 02:53:37,145 DEBUG [RS:0;jenkins-hbase20:34681-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-15 02:53:37,151 DEBUG [RS:0;jenkins-hbase20:34681-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 35658 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-15 02:53:37,154 DEBUG [RS:0;jenkins-hbase20:34681-longCompactions-0] regionserver.HStore(1912): c44270c3412c22e2c805f466aae6ff56/B is initiating minor compaction (all files) 2023-04-15 02:53:37,154 INFO [RS:0;jenkins-hbase20:34681-longCompactions-0] regionserver.HRegion(2259): Starting compaction of c44270c3412c22e2c805f466aae6ff56/B in TestAcidGuarantees,,1681527213260.c44270c3412c22e2c805f466aae6ff56. 2023-04-15 02:53:37,155 INFO [RS:0;jenkins-hbase20:34681-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/c44270c3412c22e2c805f466aae6ff56/B/e9337a469b0144028bebf63d83f2cafb, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/c44270c3412c22e2c805f466aae6ff56/B/fe072ba56cc048e29f7a020e4e4f8eea, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/c44270c3412c22e2c805f466aae6ff56/B/b9105532a32d4897a770fbf662f69001] into tmpdir=hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/c44270c3412c22e2c805f466aae6ff56/.tmp, totalSize=34.8 K 2023-04-15 02:53:37,157 DEBUG [RS:0;jenkins-hbase20:34681-longCompactions-0] compactions.Compactor(207): Compacting e9337a469b0144028bebf63d83f2cafb, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=10, earliestPutTs=1681527215603 2023-04-15 02:53:37,158 DEBUG [RS:0;jenkins-hbase20:34681-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 40338 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-15 02:53:37,159 DEBUG [RS:0;jenkins-hbase20:34681-longCompactions-0] compactions.Compactor(207): Compacting fe072ba56cc048e29f7a020e4e4f8eea, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=54, earliestPutTs=1681527215985 2023-04-15 02:53:37,167 DEBUG [RS:0;jenkins-hbase20:34681-shortCompactions-0] regionserver.HStore(1912): c44270c3412c22e2c805f466aae6ff56/A is initiating minor compaction (all files) 2023-04-15 02:53:37,167 INFO [RS:0;jenkins-hbase20:34681-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of c44270c3412c22e2c805f466aae6ff56/A in TestAcidGuarantees,,1681527213260.c44270c3412c22e2c805f466aae6ff56. 2023-04-15 02:53:37,168 INFO [RS:0;jenkins-hbase20:34681-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/c44270c3412c22e2c805f466aae6ff56/A/acacce0fff2a4181b0d8c913048e49e0, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/c44270c3412c22e2c805f466aae6ff56/A/6dd7dd3e6325495f90a18096bd9a44a8, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/c44270c3412c22e2c805f466aae6ff56/A/3bbb13c5de744a0fa342f72bc2c1221f] into tmpdir=hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/c44270c3412c22e2c805f466aae6ff56/.tmp, totalSize=39.4 K 2023-04-15 02:53:37,168 DEBUG [RS:0;jenkins-hbase20:34681-longCompactions-0] compactions.Compactor(207): Compacting b9105532a32d4897a770fbf662f69001, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=75, earliestPutTs=1681527216147 2023-04-15 02:53:37,176 DEBUG [RS:0;jenkins-hbase20:34681-shortCompactions-0] compactions.Compactor(207): Compacting acacce0fff2a4181b0d8c913048e49e0, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=10, earliestPutTs=1681527215603 2023-04-15 02:53:37,182 DEBUG [RS:0;jenkins-hbase20:34681-shortCompactions-0] compactions.Compactor(207): Compacting 6dd7dd3e6325495f90a18096bd9a44a8, keycount=250, bloomtype=ROW, size=16.2 K, encoding=NONE, compression=NONE, seqNum=54, earliestPutTs=1681527215976 2023-04-15 02:53:37,184 DEBUG [RS:0;jenkins-hbase20:34681-shortCompactions-0] compactions.Compactor(207): Compacting 3bbb13c5de744a0fa342f72bc2c1221f, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=75, earliestPutTs=1681527216147 2023-04-15 02:53:37,205 INFO [RS:0;jenkins-hbase20:34681-longCompactions-0] throttle.PressureAwareThroughputController(145): c44270c3412c22e2c805f466aae6ff56#B#compaction#9 average throughput is 1.09 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-15 02:53:37,207 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-15 02:53:37,208 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-15 02:53:37,215 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-15 02:53:37,218 INFO [RS:0;jenkins-hbase20:34681-shortCompactions-0] throttle.PressureAwareThroughputController(145): c44270c3412c22e2c805f466aae6ff56#A#compaction#10 average throughput is 1.64 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-15 02:53:37,226 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-15 02:53:37,226 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-15 02:53:37,234 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-15 02:53:37,235 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-15 02:53:37,237 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-15 02:53:37,237 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-15 02:53:37,238 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:53:37,238 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:53:37,239 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-15 02:53:37,239 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:37,239 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-15 02:53:37,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-15 02:53:37,239 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:37,239 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-15 02:53:37,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:53:37,240 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:53:37,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-15 02:53:37,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:53:37,240 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-15 02:53:37,240 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-15 02:53:37,241 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-15 02:53:37,241 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-15 02:53:37,241 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-15 02:53:37,243 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-15 02:53:37,255 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-15 02:53:37,256 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-15 02:53:37,256 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681527213260.c44270c3412c22e2c805f466aae6ff56. 2023-04-15 02:53:37,256 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681527213260.c44270c3412c22e2c805f466aae6ff56. started... 2023-04-15 02:53:37,256 INFO [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing c44270c3412c22e2c805f466aae6ff56 3/3 column families, dataSize=127.08 KB heapSize=364.15 KB 2023-04-15 02:53:37,257 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c44270c3412c22e2c805f466aae6ff56, store=A 2023-04-15 02:53:37,257 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-15 02:53:37,257 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c44270c3412c22e2c805f466aae6ff56, store=B 2023-04-15 02:53:37,257 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-15 02:53:37,257 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c44270c3412c22e2c805f466aae6ff56, store=C 2023-04-15 02:53:37,263 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-15 02:53:37,319 DEBUG [RS:0;jenkins-hbase20:34681-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/c44270c3412c22e2c805f466aae6ff56/.tmp/B/02423a6e336a4e15a0d0c1700fc4e738 as hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/c44270c3412c22e2c805f466aae6ff56/B/02423a6e336a4e15a0d0c1700fc4e738 2023-04-15 02:53:37,324 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34681] regionserver.HRegion(9158): Flush requested on c44270c3412c22e2c805f466aae6ff56 2023-04-15 02:53:37,326 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681527213260.c44270c3412c22e2c805f466aae6ff56. as already flushing 2023-04-15 02:53:37,328 DEBUG [RS:0;jenkins-hbase20:34681-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/c44270c3412c22e2c805f466aae6ff56/.tmp/A/4dee2a8a76104fdc88dd783f0a5f1aa1 as hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/c44270c3412c22e2c805f466aae6ff56/A/4dee2a8a76104fdc88dd783f0a5f1aa1 2023-04-15 02:53:37,343 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c44270c3412c22e2c805f466aae6ff56, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:53:37,344 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34681] ipc.CallRunner(144): callId: 34 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:38504 deadline: 1681527277340, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c44270c3412c22e2c805f466aae6ff56, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:37,344 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c44270c3412c22e2c805f466aae6ff56, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:53:37,344 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34681] ipc.CallRunner(144): callId: 33 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:38542 deadline: 1681527277342, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c44270c3412c22e2c805f466aae6ff56, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:37,344 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c44270c3412c22e2c805f466aae6ff56, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:53:37,345 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34681] ipc.CallRunner(144): callId: 36 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:38530 deadline: 1681527277342, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c44270c3412c22e2c805f466aae6ff56, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:37,346 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c44270c3412c22e2c805f466aae6ff56, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:53:37,349 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34681] ipc.CallRunner(144): callId: 34 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:38520 deadline: 1681527277344, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c44270c3412c22e2c805f466aae6ff56, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:37,357 INFO [RS:0;jenkins-hbase20:34681-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in c44270c3412c22e2c805f466aae6ff56/B of c44270c3412c22e2c805f466aae6ff56 into 02423a6e336a4e15a0d0c1700fc4e738(size=11.7 K), total size for store is 11.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-15 02:53:37,357 DEBUG [RS:0;jenkins-hbase20:34681-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for c44270c3412c22e2c805f466aae6ff56: 2023-04-15 02:53:37,358 INFO [RS:0;jenkins-hbase20:34681-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681527213260.c44270c3412c22e2c805f466aae6ff56., storeName=c44270c3412c22e2c805f466aae6ff56/B, priority=13, startTime=1681527217144; duration=0sec 2023-04-15 02:53:37,358 INFO [RS:0;jenkins-hbase20:34681-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in c44270c3412c22e2c805f466aae6ff56/A of c44270c3412c22e2c805f466aae6ff56 into 4dee2a8a76104fdc88dd783f0a5f1aa1(size=11.7 K), total size for store is 11.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-15 02:53:37,358 DEBUG [RS:0;jenkins-hbase20:34681-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for c44270c3412c22e2c805f466aae6ff56: 2023-04-15 02:53:37,358 INFO [RS:0;jenkins-hbase20:34681-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681527213260.c44270c3412c22e2c805f466aae6ff56., storeName=c44270c3412c22e2c805f466aae6ff56/A, priority=13, startTime=1681527217142; duration=0sec 2023-04-15 02:53:37,358 DEBUG [RS:0;jenkins-hbase20:34681-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-15 02:53:37,358 DEBUG [RS:0;jenkins-hbase20:34681-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-15 02:53:37,358 DEBUG [RS:0;jenkins-hbase20:34681-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-15 02:53:37,364 INFO [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.09 KB at sequenceid=111 (bloomFilter=true), to=hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/c44270c3412c22e2c805f466aae6ff56/.tmp/A/a7228edc07f64d3d8590a75e2dfac85a 2023-04-15 02:53:37,365 DEBUG [RS:0;jenkins-hbase20:34681-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 35658 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-15 02:53:37,365 DEBUG [RS:0;jenkins-hbase20:34681-longCompactions-0] regionserver.HStore(1912): c44270c3412c22e2c805f466aae6ff56/C is initiating minor compaction (all files) 2023-04-15 02:53:37,365 INFO [RS:0;jenkins-hbase20:34681-longCompactions-0] regionserver.HRegion(2259): Starting compaction of c44270c3412c22e2c805f466aae6ff56/C in TestAcidGuarantees,,1681527213260.c44270c3412c22e2c805f466aae6ff56. 2023-04-15 02:53:37,365 INFO [RS:0;jenkins-hbase20:34681-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/c44270c3412c22e2c805f466aae6ff56/C/081cf4a0d52047808a158eecff6304fe, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/c44270c3412c22e2c805f466aae6ff56/C/767f544ff04444b4b1e74da69ea1dc48, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/c44270c3412c22e2c805f466aae6ff56/C/0789443aa76c4c5a845f56519c2f3c41] into tmpdir=hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/c44270c3412c22e2c805f466aae6ff56/.tmp, totalSize=34.8 K 2023-04-15 02:53:37,366 DEBUG [RS:0;jenkins-hbase20:34681-longCompactions-0] compactions.Compactor(207): Compacting 081cf4a0d52047808a158eecff6304fe, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=10, earliestPutTs=1681527215603 2023-04-15 02:53:37,367 DEBUG [RS:0;jenkins-hbase20:34681-longCompactions-0] compactions.Compactor(207): Compacting 767f544ff04444b4b1e74da69ea1dc48, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=54, earliestPutTs=1681527215985 2023-04-15 02:53:37,368 DEBUG [RS:0;jenkins-hbase20:34681-longCompactions-0] compactions.Compactor(207): Compacting 0789443aa76c4c5a845f56519c2f3c41, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=75, earliestPutTs=1681527216147 2023-04-15 02:53:37,387 INFO [RS:0;jenkins-hbase20:34681-longCompactions-0] throttle.PressureAwareThroughputController(145): c44270c3412c22e2c805f466aae6ff56#C#compaction#13 average throughput is 1.64 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-15 02:53:37,400 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c44270c3412c22e2c805f466aae6ff56, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:53:37,405 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34681] ipc.CallRunner(144): callId: 35 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:38540 deadline: 1681527277362, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c44270c3412c22e2c805f466aae6ff56, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:37,419 INFO [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=111 (bloomFilter=true), to=hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/c44270c3412c22e2c805f466aae6ff56/.tmp/B/c2cf376d42b84a2cb7ec7a3735e47c7d 2023-04-15 02:53:37,440 DEBUG [RS:0;jenkins-hbase20:34681-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/c44270c3412c22e2c805f466aae6ff56/.tmp/C/d3fb7497c5d54289bef7742bb9d41db6 as hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/c44270c3412c22e2c805f466aae6ff56/C/d3fb7497c5d54289bef7742bb9d41db6 2023-04-15 02:53:37,447 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c44270c3412c22e2c805f466aae6ff56, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:53:37,447 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34681] ipc.CallRunner(144): callId: 35 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:38542 deadline: 1681527277447, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c44270c3412c22e2c805f466aae6ff56, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:37,448 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c44270c3412c22e2c805f466aae6ff56, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:53:37,448 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34681] ipc.CallRunner(144): callId: 36 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:38504 deadline: 1681527277447, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c44270c3412c22e2c805f466aae6ff56, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:37,449 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c44270c3412c22e2c805f466aae6ff56, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:53:37,449 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34681] ipc.CallRunner(144): callId: 38 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:38530 deadline: 1681527277448, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c44270c3412c22e2c805f466aae6ff56, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:37,453 INFO [RS:0;jenkins-hbase20:34681-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in c44270c3412c22e2c805f466aae6ff56/C of c44270c3412c22e2c805f466aae6ff56 into d3fb7497c5d54289bef7742bb9d41db6(size=11.7 K), total size for store is 11.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-15 02:53:37,455 DEBUG [RS:0;jenkins-hbase20:34681-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for c44270c3412c22e2c805f466aae6ff56: 2023-04-15 02:53:37,453 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c44270c3412c22e2c805f466aae6ff56, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:53:37,456 INFO [RS:0;jenkins-hbase20:34681-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681527213260.c44270c3412c22e2c805f466aae6ff56., storeName=c44270c3412c22e2c805f466aae6ff56/C, priority=13, startTime=1681527217144; duration=0sec 2023-04-15 02:53:37,456 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34681] ipc.CallRunner(144): callId: 36 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:38520 deadline: 1681527277453, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c44270c3412c22e2c805f466aae6ff56, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:37,457 DEBUG [RS:0;jenkins-hbase20:34681-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-15 02:53:37,486 INFO [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=111 (bloomFilter=true), to=hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/c44270c3412c22e2c805f466aae6ff56/.tmp/C/fe70e4bee82f4087938046976ab556b6 2023-04-15 02:53:37,496 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/c44270c3412c22e2c805f466aae6ff56/.tmp/A/a7228edc07f64d3d8590a75e2dfac85a as hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/c44270c3412c22e2c805f466aae6ff56/A/a7228edc07f64d3d8590a75e2dfac85a 2023-04-15 02:53:37,506 INFO [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/c44270c3412c22e2c805f466aae6ff56/A/a7228edc07f64d3d8590a75e2dfac85a, entries=150, sequenceid=111, filesize=11.6 K 2023-04-15 02:53:37,507 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/c44270c3412c22e2c805f466aae6ff56/.tmp/B/c2cf376d42b84a2cb7ec7a3735e47c7d as hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/c44270c3412c22e2c805f466aae6ff56/B/c2cf376d42b84a2cb7ec7a3735e47c7d 2023-04-15 02:53:37,509 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c44270c3412c22e2c805f466aae6ff56, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:53:37,513 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34681] ipc.CallRunner(144): callId: 37 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:38540 deadline: 1681527277507, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c44270c3412c22e2c805f466aae6ff56, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:37,521 INFO [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/c44270c3412c22e2c805f466aae6ff56/B/c2cf376d42b84a2cb7ec7a3735e47c7d, entries=150, sequenceid=111, filesize=11.6 K 2023-04-15 02:53:37,525 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/c44270c3412c22e2c805f466aae6ff56/.tmp/C/fe70e4bee82f4087938046976ab556b6 as hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/c44270c3412c22e2c805f466aae6ff56/C/fe70e4bee82f4087938046976ab556b6 2023-04-15 02:53:37,537 INFO [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/c44270c3412c22e2c805f466aae6ff56/C/fe70e4bee82f4087938046976ab556b6, entries=150, sequenceid=111, filesize=11.6 K 2023-04-15 02:53:37,540 INFO [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~127.08 KB/130125, heapSize ~364.10 KB/372840, currentSize=60.38 KB/61830 for c44270c3412c22e2c805f466aae6ff56 in 284ms, sequenceid=111, compaction requested=false 2023-04-15 02:53:37,540 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for c44270c3412c22e2c805f466aae6ff56: 2023-04-15 02:53:37,540 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681527213260.c44270c3412c22e2c805f466aae6ff56. 2023-04-15 02:53:37,540 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-15 02:53:37,542 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-15 02:53:37,542 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:37,542 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-15 02:53:37,542 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34681,1681527209133' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-15 02:53:37,544 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:37,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:37,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:53:37,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:53:37,544 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:53:37,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:53:37,545 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:53:37,545 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-15 02:53:37,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:53:37,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:53:37,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:37,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:53:37,548 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34681,1681527209133' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-15 02:53:37,548 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4edb6ebb[Count = 0] remaining members to acquire global barrier 2023-04-15 02:53:37,548 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-15 02:53:37,548 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:53:37,549 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:53:37,549 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:53:37,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:53:37,549 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-15 02:53:37,550 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-15 02:53:37,550 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34681,1681527209133' in zk 2023-04-15 02:53:37,550 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:37,550 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-15 02:53:37,551 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-15 02:53:37,551 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-15 02:53:37,551 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-15 02:53:37,553 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:37,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:37,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:53:37,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:53:37,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:53:37,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:53:37,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:53:37,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:37,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:53:37,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:53:37,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:37,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34681,1681527209133': 2023-04-15 02:53:37,559 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34681,1681527209133' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-15 02:53:37,559 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-15 02:53:37,559 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-15 02:53:37,559 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-15 02:53:37,559 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-15 02:53:37,559 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-15 02:53:37,565 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:53:37,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:53:37,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:53:37,566 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:53:37,566 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-15 02:53:37,566 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:53:37,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:53:37,567 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-15 02:53:37,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-15 02:53:37,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:53:37,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:53:37,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:53:37,570 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:37,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:53:37,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:53:37,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:53:37,572 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:37,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:37,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:53:37,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:53:37,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:37,575 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-15 02:53:37,575 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:37,575 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-15 02:53:37,575 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:53:37,575 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-15 02:53:37,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-15 02:53:37,575 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-15 02:53:37,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-15 02:53:37,576 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:53:37,576 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:37,576 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:53:37,576 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:53:37,576 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:53:37,576 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-15 02:53:37,576 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-15 02:53:37,577 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-15 02:53:37,578 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-15 02:53:37,578 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-15 02:53:37,578 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-15 02:53:37,653 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34681] regionserver.HRegion(9158): Flush requested on c44270c3412c22e2c805f466aae6ff56 2023-04-15 02:53:37,653 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing c44270c3412c22e2c805f466aae6ff56 3/3 column families, dataSize=67.09 KB heapSize=199.97 KB 2023-04-15 02:53:37,653 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c44270c3412c22e2c805f466aae6ff56, store=A 2023-04-15 02:53:37,653 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-15 02:53:37,653 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c44270c3412c22e2c805f466aae6ff56, store=B 2023-04-15 02:53:37,653 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-15 02:53:37,654 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c44270c3412c22e2c805f466aae6ff56, store=C 2023-04-15 02:53:37,654 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-15 02:53:37,678 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-15 02:53:37,679 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-15 02:53:37,685 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-15 02:53:37,696 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-15 02:53:37,697 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-15 02:53:37,697 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-15 02:53:37,697 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-15 02:53:37,698 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-15 02:53:37,698 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-15 02:53:37,699 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:53:37,699 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:53:37,700 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c44270c3412c22e2c805f466aae6ff56, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:53:37,700 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34681] ipc.CallRunner(144): callId: 44 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:38530 deadline: 1681527277685, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c44270c3412c22e2c805f466aae6ff56, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:37,701 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-15 02:53:37,701 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:37,701 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-15 02:53:37,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-15 02:53:37,701 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:37,701 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-15 02:53:37,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:53:37,702 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:53:37,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-15 02:53:37,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:53:37,702 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-15 02:53:37,702 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c44270c3412c22e2c805f466aae6ff56, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:53:37,703 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34681] ipc.CallRunner(144): callId: 42 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:38542 deadline: 1681527277696, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c44270c3412c22e2c805f466aae6ff56, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:37,703 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c44270c3412c22e2c805f466aae6ff56, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:53:37,704 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34681] ipc.CallRunner(144): callId: 41 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:38520 deadline: 1681527277699, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c44270c3412c22e2c805f466aae6ff56, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:37,704 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c44270c3412c22e2c805f466aae6ff56, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:53:37,705 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-15 02:53:37,705 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34681] ipc.CallRunner(144): callId: 43 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:38504 deadline: 1681527277700, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c44270c3412c22e2c805f466aae6ff56, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:37,705 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-15 02:53:37,705 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-15 02:53:37,705 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-15 02:53:37,706 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-15 02:53:37,706 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-15 02:53:37,706 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-15 02:53:37,706 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681527213260.c44270c3412c22e2c805f466aae6ff56. 2023-04-15 02:53:37,706 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681527213260.c44270c3412c22e2c805f466aae6ff56. started... 2023-04-15 02:53:37,706 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681527213260.c44270c3412c22e2c805f466aae6ff56. as already flushing 2023-04-15 02:53:37,707 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681527213260.c44270c3412c22e2c805f466aae6ff56. 2023-04-15 02:53:37,707 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-15 02:53:37,707 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-15 02:53:37,707 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:37,707 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-15 02:53:37,707 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34681,1681527209133' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-15 02:53:37,710 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:37,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:37,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:53:37,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:53:37,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:53:37,710 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:53:37,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:53:37,712 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:53:37,712 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-15 02:53:37,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:53:37,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:37,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:53:37,713 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34681,1681527209133' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-15 02:53:37,713 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@12fc0859[Count = 0] remaining members to acquire global barrier 2023-04-15 02:53:37,713 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-15 02:53:37,713 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:53:37,714 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:53:37,715 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:53:37,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:53:37,715 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-15 02:53:37,715 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-15 02:53:37,715 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34681,1681527209133' in zk 2023-04-15 02:53:37,715 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:37,715 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-15 02:53:37,716 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c44270c3412c22e2c805f466aae6ff56, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:53:37,717 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34681] ipc.CallRunner(144): callId: 39 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:38540 deadline: 1681527277716, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c44270c3412c22e2c805f466aae6ff56, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:37,717 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:37,717 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-15 02:53:37,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:37,717 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-15 02:53:37,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:53:37,717 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-15 02:53:37,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:53:37,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:53:37,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:53:37,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:53:37,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:37,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:53:37,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:53:37,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:37,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34681,1681527209133': 2023-04-15 02:53:37,724 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34681,1681527209133' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-15 02:53:37,724 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-15 02:53:37,724 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-15 02:53:37,724 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-15 02:53:37,724 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-15 02:53:37,724 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-15 02:53:37,728 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:53:37,728 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-15 02:53:37,728 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:53:37,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:53:37,729 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:53:37,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:53:37,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:53:37,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:53:37,730 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-15 02:53:37,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-15 02:53:37,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:53:37,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:53:37,732 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:37,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:53:37,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:53:37,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:53:37,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:37,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:53:37,734 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:37,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:53:37,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:37,741 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-15 02:53:37,741 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-15 02:53:37,741 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:37,741 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:53:37,741 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-15 02:53:37,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-15 02:53:37,741 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:53:37,742 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:37,742 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:53:37,742 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-15 02:53:37,742 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-15 02:53:37,742 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-15 02:53:37,742 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-15 02:53:37,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-15 02:53:37,743 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-15 02:53:37,743 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-15 02:53:37,742 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:53:37,743 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:53:37,805 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c44270c3412c22e2c805f466aae6ff56, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:53:37,805 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34681] ipc.CallRunner(144): callId: 46 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:38530 deadline: 1681527277802, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c44270c3412c22e2c805f466aae6ff56, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:37,805 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c44270c3412c22e2c805f466aae6ff56, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:53:37,806 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34681] ipc.CallRunner(144): callId: 44 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:38542 deadline: 1681527277805, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c44270c3412c22e2c805f466aae6ff56, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:37,808 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c44270c3412c22e2c805f466aae6ff56, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:53:37,808 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34681] ipc.CallRunner(144): callId: 43 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:38520 deadline: 1681527277805, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c44270c3412c22e2c805f466aae6ff56, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:37,808 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c44270c3412c22e2c805f466aae6ff56, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:53:37,809 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34681] ipc.CallRunner(144): callId: 45 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:38504 deadline: 1681527277807, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c44270c3412c22e2c805f466aae6ff56, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:37,843 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-15 02:53:37,844 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-15 02:53:37,847 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-15 02:53:37,850 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-15 02:53:37,850 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-15 02:53:37,850 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-15 02:53:37,850 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-15 02:53:37,856 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-15 02:53:37,857 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-15 02:53:37,858 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:53:37,858 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:53:37,859 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-15 02:53:37,859 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:37,859 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-15 02:53:37,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-15 02:53:37,860 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:37,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:53:37,860 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-15 02:53:37,860 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:53:37,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-15 02:53:37,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:53:37,861 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-15 02:53:37,862 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-15 02:53:37,862 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-15 02:53:37,862 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-15 02:53:37,862 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-15 02:53:37,863 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-15 02:53:37,863 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-15 02:53:37,863 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-15 02:53:37,863 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681527213260.c44270c3412c22e2c805f466aae6ff56. 2023-04-15 02:53:37,863 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681527213260.c44270c3412c22e2c805f466aae6ff56. started... 2023-04-15 02:53:37,863 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681527213260.c44270c3412c22e2c805f466aae6ff56. as already flushing 2023-04-15 02:53:37,863 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681527213260.c44270c3412c22e2c805f466aae6ff56. 2023-04-15 02:53:37,863 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-15 02:53:37,864 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-15 02:53:37,864 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:37,864 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-15 02:53:37,864 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34681,1681527209133' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-15 02:53:37,877 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:37,877 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:53:37,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:37,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:53:37,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:53:37,878 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:53:37,878 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-15 02:53:37,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:53:37,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:53:37,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:53:37,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:37,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:53:37,881 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34681,1681527209133' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-15 02:53:37,881 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3bbfdaa4[Count = 0] remaining members to acquire global barrier 2023-04-15 02:53:37,881 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-15 02:53:37,881 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:53:37,883 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:53:37,883 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:53:37,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:53:37,884 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-15 02:53:37,884 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-15 02:53:37,884 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34681,1681527209133' in zk 2023-04-15 02:53:37,884 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:37,884 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-15 02:53:37,885 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-15 02:53:37,885 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-15 02:53:37,885 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-15 02:53:37,890 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:37,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:37,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:53:37,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:53:37,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:53:37,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:53:37,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:53:37,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:37,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:53:37,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:53:37,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:37,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34681,1681527209133': 2023-04-15 02:53:37,895 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34681,1681527209133' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-15 02:53:37,895 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-15 02:53:37,895 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-15 02:53:37,895 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-15 02:53:37,895 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-15 02:53:37,895 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-15 02:53:37,898 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:53:37,899 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-15 02:53:37,899 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:53:37,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:53:37,898 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:53:37,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:53:37,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:53:37,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:53:37,899 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-15 02:53:37,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-15 02:53:37,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:53:37,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:53:37,900 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:37,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:53:37,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:53:37,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:53:37,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:37,903 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:37,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:53:37,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:53:37,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:37,911 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:37,911 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:53:37,911 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:53:37,911 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:37,911 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:53:37,911 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:53:37,911 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:53:37,911 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-15 02:53:37,911 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-15 02:53:37,911 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-15 02:53:37,911 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-15 02:53:37,912 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-15 02:53:37,912 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-15 02:53:37,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-15 02:53:37,913 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-15 02:53:37,913 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-15 02:53:37,913 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-15 02:53:37,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-15 02:53:38,008 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c44270c3412c22e2c805f466aae6ff56, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:53:38,009 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34681] ipc.CallRunner(144): callId: 46 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:38542 deadline: 1681527278008, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c44270c3412c22e2c805f466aae6ff56, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:38,012 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c44270c3412c22e2c805f466aae6ff56, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:53:38,012 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34681] ipc.CallRunner(144): callId: 48 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:38530 deadline: 1681527278010, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c44270c3412c22e2c805f466aae6ff56, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:38,013 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c44270c3412c22e2c805f466aae6ff56, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:53:38,013 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-15 02:53:38,013 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34681] ipc.CallRunner(144): callId: 45 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:38520 deadline: 1681527278011, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c44270c3412c22e2c805f466aae6ff56, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:38,014 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-15 02:53:38,017 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-15 02:53:38,018 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c44270c3412c22e2c805f466aae6ff56, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:53:38,020 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c44270c3412c22e2c805f466aae6ff56, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:53:38,020 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34681] ipc.CallRunner(144): callId: 41 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:38540 deadline: 1681527278019, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c44270c3412c22e2c805f466aae6ff56, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:38,020 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34681] ipc.CallRunner(144): callId: 47 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:38504 deadline: 1681527278014, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c44270c3412c22e2c805f466aae6ff56, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:38,032 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-15 02:53:38,032 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-15 02:53:38,032 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-15 02:53:38,033 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-15 02:53:38,033 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-15 02:53:38,033 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-15 02:53:38,034 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:53:38,034 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:53:38,035 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:38,035 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-15 02:53:38,035 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-15 02:53:38,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-15 02:53:38,035 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:38,036 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-15 02:53:38,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:53:38,036 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:53:38,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-15 02:53:38,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:53:38,037 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-15 02:53:38,041 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-15 02:53:38,042 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-15 02:53:38,042 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-15 02:53:38,042 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-15 02:53:38,044 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-15 02:53:38,044 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-15 02:53:38,044 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-15 02:53:38,044 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681527213260.c44270c3412c22e2c805f466aae6ff56. 2023-04-15 02:53:38,044 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681527213260.c44270c3412c22e2c805f466aae6ff56. started... 2023-04-15 02:53:38,045 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681527213260.c44270c3412c22e2c805f466aae6ff56. as already flushing 2023-04-15 02:53:38,045 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681527213260.c44270c3412c22e2c805f466aae6ff56. 2023-04-15 02:53:38,045 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-15 02:53:38,045 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-15 02:53:38,045 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:38,045 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-15 02:53:38,045 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34681,1681527209133' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-15 02:53:38,052 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:38,052 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:53:38,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:38,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:53:38,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:53:38,053 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:53:38,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:53:38,053 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-15 02:53:38,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:53:38,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:53:38,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:38,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:53:38,055 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34681,1681527209133' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-15 02:53:38,055 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6da9ab98[Count = 0] remaining members to acquire global barrier 2023-04-15 02:53:38,055 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-15 02:53:38,055 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:53:38,057 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:53:38,057 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:53:38,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:53:38,057 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:38,057 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-15 02:53:38,057 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-15 02:53:38,057 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-15 02:53:38,057 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34681,1681527209133' in zk 2023-04-15 02:53:38,059 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-15 02:53:38,059 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:38,059 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-15 02:53:38,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:38,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:53:38,059 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-15 02:53:38,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:53:38,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:53:38,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:53:38,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:53:38,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:38,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:53:38,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:53:38,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:38,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34681,1681527209133': 2023-04-15 02:53:38,066 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34681,1681527209133' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-15 02:53:38,066 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-15 02:53:38,066 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-15 02:53:38,066 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-15 02:53:38,066 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-15 02:53:38,066 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-15 02:53:38,073 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:53:38,074 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:53:38,074 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:53:38,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:53:38,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:53:38,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:53:38,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:53:38,074 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-15 02:53:38,074 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-15 02:53:38,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-15 02:53:38,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:53:38,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:53:38,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:53:38,075 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:38,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:53:38,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:53:38,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:38,077 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:38,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:53:38,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:53:38,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:38,082 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:38,082 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-15 02:53:38,082 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:53:38,082 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-15 02:53:38,082 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-15 02:53:38,082 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-15 02:53:38,082 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-15 02:53:38,082 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-15 02:53:38,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-15 02:53:38,082 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:53:38,082 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:38,082 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-15 02:53:38,082 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:53:38,082 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-15 02:53:38,083 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:53:38,083 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-15 02:53:38,083 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:53:38,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-15 02:53:38,098 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=127 (bloomFilter=true), to=hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/c44270c3412c22e2c805f466aae6ff56/.tmp/A/1f795105d9aa47a4b295f5e05d7a5e03 2023-04-15 02:53:38,132 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=127 (bloomFilter=true), to=hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/c44270c3412c22e2c805f466aae6ff56/.tmp/B/a51d0e04a11143799840143be80d3d30 2023-04-15 02:53:38,160 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=127 (bloomFilter=true), to=hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/c44270c3412c22e2c805f466aae6ff56/.tmp/C/bd10531039a0472b8dd505a7b846eabc 2023-04-15 02:53:38,168 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/c44270c3412c22e2c805f466aae6ff56/.tmp/A/1f795105d9aa47a4b295f5e05d7a5e03 as hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/c44270c3412c22e2c805f466aae6ff56/A/1f795105d9aa47a4b295f5e05d7a5e03 2023-04-15 02:53:38,178 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/c44270c3412c22e2c805f466aae6ff56/A/1f795105d9aa47a4b295f5e05d7a5e03, entries=150, sequenceid=127, filesize=11.6 K 2023-04-15 02:53:38,182 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/c44270c3412c22e2c805f466aae6ff56/.tmp/B/a51d0e04a11143799840143be80d3d30 as hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/c44270c3412c22e2c805f466aae6ff56/B/a51d0e04a11143799840143be80d3d30 2023-04-15 02:53:38,183 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-15 02:53:38,187 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-15 02:53:38,190 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-15 02:53:38,192 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-15 02:53:38,192 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-15 02:53:38,192 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-15 02:53:38,192 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-15 02:53:38,193 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-15 02:53:38,194 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-15 02:53:38,194 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:53:38,194 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:53:38,195 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-15 02:53:38,195 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:38,195 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-15 02:53:38,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-15 02:53:38,196 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:38,196 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-15 02:53:38,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:53:38,196 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:53:38,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-15 02:53:38,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:53:38,197 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-15 02:53:38,197 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/c44270c3412c22e2c805f466aae6ff56/B/a51d0e04a11143799840143be80d3d30, entries=150, sequenceid=127, filesize=11.6 K 2023-04-15 02:53:38,197 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-15 02:53:38,197 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-15 02:53:38,197 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-15 02:53:38,197 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-15 02:53:38,198 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-15 02:53:38,199 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-15 02:53:38,199 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-15 02:53:38,199 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681527213260.c44270c3412c22e2c805f466aae6ff56. 2023-04-15 02:53:38,199 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681527213260.c44270c3412c22e2c805f466aae6ff56. started... 2023-04-15 02:53:38,201 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681527213260.c44270c3412c22e2c805f466aae6ff56. as already flushing 2023-04-15 02:53:38,201 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681527213260.c44270c3412c22e2c805f466aae6ff56. 2023-04-15 02:53:38,201 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-15 02:53:38,201 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-15 02:53:38,201 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:38,201 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-15 02:53:38,201 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34681,1681527209133' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-15 02:53:38,203 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/c44270c3412c22e2c805f466aae6ff56/.tmp/C/bd10531039a0472b8dd505a7b846eabc as hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/c44270c3412c22e2c805f466aae6ff56/C/bd10531039a0472b8dd505a7b846eabc 2023-04-15 02:53:38,207 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:53:38,207 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:38,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:38,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:53:38,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:53:38,207 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:53:38,207 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-15 02:53:38,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:53:38,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:53:38,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:53:38,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:38,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:53:38,210 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34681,1681527209133' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-15 02:53:38,210 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7812be20[Count = 0] remaining members to acquire global barrier 2023-04-15 02:53:38,210 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-15 02:53:38,210 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:53:38,211 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:53:38,211 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:53:38,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:53:38,212 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-15 02:53:38,212 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-15 02:53:38,212 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34681,1681527209133' in zk 2023-04-15 02:53:38,212 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:38,212 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-15 02:53:38,213 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-15 02:53:38,213 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-15 02:53:38,213 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-15 02:53:38,214 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/c44270c3412c22e2c805f466aae6ff56/C/bd10531039a0472b8dd505a7b846eabc, entries=150, sequenceid=127, filesize=11.6 K 2023-04-15 02:53:38,214 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:38,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:38,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:53:38,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:53:38,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:53:38,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:53:38,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:53:38,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:38,217 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~67.09 KB/68700, heapSize ~199.92 KB/204720, currentSize=107.34 KB/109920 for c44270c3412c22e2c805f466aae6ff56 in 564ms, sequenceid=127, compaction requested=true 2023-04-15 02:53:38,217 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for c44270c3412c22e2c805f466aae6ff56: 2023-04-15 02:53:38,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:53:38,217 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-15 02:53:38,217 DEBUG [RS:0;jenkins-hbase20:34681-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-15 02:53:38,218 DEBUG [RS:0;jenkins-hbase20:34681-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-15 02:53:38,218 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-15 02:53:38,218 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-15 02:53:38,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:53:38,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:38,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34681,1681527209133': 2023-04-15 02:53:38,220 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34681,1681527209133' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-15 02:53:38,220 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-15 02:53:38,220 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-15 02:53:38,220 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-15 02:53:38,220 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-15 02:53:38,220 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-15 02:53:38,221 DEBUG [RS:0;jenkins-hbase20:34681-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 35761 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-15 02:53:38,221 DEBUG [RS:0;jenkins-hbase20:34681-shortCompactions-0] regionserver.HStore(1912): c44270c3412c22e2c805f466aae6ff56/A is initiating minor compaction (all files) 2023-04-15 02:53:38,221 INFO [RS:0;jenkins-hbase20:34681-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of c44270c3412c22e2c805f466aae6ff56/A in TestAcidGuarantees,,1681527213260.c44270c3412c22e2c805f466aae6ff56. 2023-04-15 02:53:38,221 INFO [RS:0;jenkins-hbase20:34681-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/c44270c3412c22e2c805f466aae6ff56/A/4dee2a8a76104fdc88dd783f0a5f1aa1, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/c44270c3412c22e2c805f466aae6ff56/A/a7228edc07f64d3d8590a75e2dfac85a, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/c44270c3412c22e2c805f466aae6ff56/A/1f795105d9aa47a4b295f5e05d7a5e03] into tmpdir=hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/c44270c3412c22e2c805f466aae6ff56/.tmp, totalSize=34.9 K 2023-04-15 02:53:38,221 DEBUG [RS:0;jenkins-hbase20:34681-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 35761 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-15 02:53:38,221 DEBUG [RS:0;jenkins-hbase20:34681-longCompactions-0] regionserver.HStore(1912): c44270c3412c22e2c805f466aae6ff56/B is initiating minor compaction (all files) 2023-04-15 02:53:38,221 INFO [RS:0;jenkins-hbase20:34681-longCompactions-0] regionserver.HRegion(2259): Starting compaction of c44270c3412c22e2c805f466aae6ff56/B in TestAcidGuarantees,,1681527213260.c44270c3412c22e2c805f466aae6ff56. 2023-04-15 02:53:38,222 INFO [RS:0;jenkins-hbase20:34681-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/c44270c3412c22e2c805f466aae6ff56/B/02423a6e336a4e15a0d0c1700fc4e738, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/c44270c3412c22e2c805f466aae6ff56/B/c2cf376d42b84a2cb7ec7a3735e47c7d, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/c44270c3412c22e2c805f466aae6ff56/B/a51d0e04a11143799840143be80d3d30] into tmpdir=hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/c44270c3412c22e2c805f466aae6ff56/.tmp, totalSize=34.9 K 2023-04-15 02:53:38,223 DEBUG [RS:0;jenkins-hbase20:34681-shortCompactions-0] compactions.Compactor(207): Compacting 4dee2a8a76104fdc88dd783f0a5f1aa1, keycount=150, bloomtype=ROW, size=11.7 K, encoding=NONE, compression=NONE, seqNum=75, earliestPutTs=1681527216147 2023-04-15 02:53:38,225 DEBUG [RS:0;jenkins-hbase20:34681-longCompactions-0] compactions.Compactor(207): Compacting 02423a6e336a4e15a0d0c1700fc4e738, keycount=150, bloomtype=ROW, size=11.7 K, encoding=NONE, compression=NONE, seqNum=75, earliestPutTs=1681527216147 2023-04-15 02:53:38,226 DEBUG [RS:0;jenkins-hbase20:34681-shortCompactions-0] compactions.Compactor(207): Compacting a7228edc07f64d3d8590a75e2dfac85a, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=111, earliestPutTs=1681527216980 2023-04-15 02:53:38,227 DEBUG [RS:0;jenkins-hbase20:34681-longCompactions-0] compactions.Compactor(207): Compacting c2cf376d42b84a2cb7ec7a3735e47c7d, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=111, earliestPutTs=1681527216980 2023-04-15 02:53:38,227 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:53:38,227 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-15 02:53:38,227 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:53:38,227 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:53:38,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:53:38,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:53:38,227 DEBUG [RS:0;jenkins-hbase20:34681-shortCompactions-0] compactions.Compactor(207): Compacting 1f795105d9aa47a4b295f5e05d7a5e03, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=127, earliestPutTs=1681527217339 2023-04-15 02:53:38,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:53:38,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:53:38,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:53:38,228 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-15 02:53:38,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-15 02:53:38,228 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:38,229 DEBUG [RS:0;jenkins-hbase20:34681-longCompactions-0] compactions.Compactor(207): Compacting a51d0e04a11143799840143be80d3d30, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=127, earliestPutTs=1681527217339 2023-04-15 02:53:38,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:53:38,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:53:38,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:53:38,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:53:38,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:38,230 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:38,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:53:38,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:53:38,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:38,236 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-15 02:53:38,236 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-15 02:53:38,236 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-15 02:53:38,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-15 02:53:38,237 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-15 02:53:38,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-15 02:53:38,237 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:38,237 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:53:38,237 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:53:38,237 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:38,237 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:53:38,237 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:53:38,237 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-15 02:53:38,237 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-15 02:53:38,237 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-15 02:53:38,237 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:53:38,238 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-15 02:53:38,238 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-15 02:53:38,237 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-15 02:53:38,248 INFO [RS:0;jenkins-hbase20:34681-shortCompactions-0] throttle.PressureAwareThroughputController(145): c44270c3412c22e2c805f466aae6ff56#A#compaction#18 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-15 02:53:38,249 INFO [RS:0;jenkins-hbase20:34681-longCompactions-0] throttle.PressureAwareThroughputController(145): c44270c3412c22e2c805f466aae6ff56#B#compaction#19 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-15 02:53:38,295 DEBUG [RS:0;jenkins-hbase20:34681-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/c44270c3412c22e2c805f466aae6ff56/.tmp/A/c6471c87fd14400580d23d2501310834 as hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/c44270c3412c22e2c805f466aae6ff56/A/c6471c87fd14400580d23d2501310834 2023-04-15 02:53:38,306 INFO [RS:0;jenkins-hbase20:34681-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in c44270c3412c22e2c805f466aae6ff56/A of c44270c3412c22e2c805f466aae6ff56 into c6471c87fd14400580d23d2501310834(size=11.8 K), total size for store is 11.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-15 02:53:38,306 DEBUG [RS:0;jenkins-hbase20:34681-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for c44270c3412c22e2c805f466aae6ff56: 2023-04-15 02:53:38,306 INFO [RS:0;jenkins-hbase20:34681-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681527213260.c44270c3412c22e2c805f466aae6ff56., storeName=c44270c3412c22e2c805f466aae6ff56/A, priority=13, startTime=1681527218217; duration=0sec 2023-04-15 02:53:38,308 DEBUG [RS:0;jenkins-hbase20:34681-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-15 02:53:38,308 DEBUG [RS:0;jenkins-hbase20:34681-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-15 02:53:38,310 DEBUG [RS:0;jenkins-hbase20:34681-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 35761 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-15 02:53:38,310 DEBUG [RS:0;jenkins-hbase20:34681-shortCompactions-0] regionserver.HStore(1912): c44270c3412c22e2c805f466aae6ff56/C is initiating minor compaction (all files) 2023-04-15 02:53:38,310 INFO [RS:0;jenkins-hbase20:34681-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of c44270c3412c22e2c805f466aae6ff56/C in TestAcidGuarantees,,1681527213260.c44270c3412c22e2c805f466aae6ff56. 2023-04-15 02:53:38,310 INFO [RS:0;jenkins-hbase20:34681-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/c44270c3412c22e2c805f466aae6ff56/C/d3fb7497c5d54289bef7742bb9d41db6, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/c44270c3412c22e2c805f466aae6ff56/C/fe70e4bee82f4087938046976ab556b6, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/c44270c3412c22e2c805f466aae6ff56/C/bd10531039a0472b8dd505a7b846eabc] into tmpdir=hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/c44270c3412c22e2c805f466aae6ff56/.tmp, totalSize=34.9 K 2023-04-15 02:53:38,311 DEBUG [RS:0;jenkins-hbase20:34681-shortCompactions-0] compactions.Compactor(207): Compacting d3fb7497c5d54289bef7742bb9d41db6, keycount=150, bloomtype=ROW, size=11.7 K, encoding=NONE, compression=NONE, seqNum=75, earliestPutTs=1681527216147 2023-04-15 02:53:38,312 DEBUG [RS:0;jenkins-hbase20:34681-shortCompactions-0] compactions.Compactor(207): Compacting fe70e4bee82f4087938046976ab556b6, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=111, earliestPutTs=1681527216980 2023-04-15 02:53:38,313 DEBUG [RS:0;jenkins-hbase20:34681-shortCompactions-0] compactions.Compactor(207): Compacting bd10531039a0472b8dd505a7b846eabc, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=127, earliestPutTs=1681527217339 2023-04-15 02:53:38,315 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34681] regionserver.HRegion(9158): Flush requested on c44270c3412c22e2c805f466aae6ff56 2023-04-15 02:53:38,316 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing c44270c3412c22e2c805f466aae6ff56 3/3 column families, dataSize=114.05 KB heapSize=339.42 KB 2023-04-15 02:53:38,317 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c44270c3412c22e2c805f466aae6ff56, store=A 2023-04-15 02:53:38,317 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-15 02:53:38,317 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c44270c3412c22e2c805f466aae6ff56, store=B 2023-04-15 02:53:38,317 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-15 02:53:38,317 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c44270c3412c22e2c805f466aae6ff56, store=C 2023-04-15 02:53:38,317 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-15 02:53:38,339 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-15 02:53:38,339 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-15 02:53:38,342 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-15 02:53:38,345 DEBUG [HBase-Metrics2-1] impl.GlobalMetricRegistriesAdapter(130): Registering adapter for the MetricRegistry: RegionServer,sub=Coprocessor.Region.CP_org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint 2023-04-15 02:53:38,347 INFO [HBase-Metrics2-1] impl.GlobalMetricRegistriesAdapter(134): Registering RegionServer,sub=Coprocessor.Region.CP_org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint Metrics about HBase RegionObservers 2023-04-15 02:53:38,347 INFO [RS:0;jenkins-hbase20:34681-shortCompactions-0] throttle.PressureAwareThroughputController(145): c44270c3412c22e2c805f466aae6ff56#C#compaction#21 average throughput is 2.18 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-15 02:53:38,350 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-15 02:53:38,355 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=148 (bloomFilter=true), to=hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/c44270c3412c22e2c805f466aae6ff56/.tmp/A/94a6ea33c5964300bba31cc9d9175e53 2023-04-15 02:53:38,357 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-15 02:53:38,357 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-15 02:53:38,357 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-15 02:53:38,358 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-15 02:53:38,358 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-15 02:53:38,358 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c44270c3412c22e2c805f466aae6ff56, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:53:38,359 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:53:38,359 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:53:38,359 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34681] ipc.CallRunner(144): callId: 51 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:38542 deadline: 1681527278348, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c44270c3412c22e2c805f466aae6ff56, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:38,361 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c44270c3412c22e2c805f466aae6ff56, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:53:38,361 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-15 02:53:38,361 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:38,361 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-15 02:53:38,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-15 02:53:38,361 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34681] ipc.CallRunner(144): callId: 50 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:38520 deadline: 1681527278348, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c44270c3412c22e2c805f466aae6ff56, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:38,363 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:38,363 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-15 02:53:38,363 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c44270c3412c22e2c805f466aae6ff56, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:53:38,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:53:38,364 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34681] ipc.CallRunner(144): callId: 52 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:38530 deadline: 1681527278349, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c44270c3412c22e2c805f466aae6ff56, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:38,365 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:53:38,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-15 02:53:38,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:53:38,365 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-15 02:53:38,367 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c44270c3412c22e2c805f466aae6ff56, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:53:38,367 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34681] ipc.CallRunner(144): callId: 51 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:38504 deadline: 1681527278359, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c44270c3412c22e2c805f466aae6ff56, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:38,378 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-15 02:53:38,378 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-15 02:53:38,378 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-15 02:53:38,378 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-15 02:53:38,393 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-15 02:53:38,393 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-15 02:53:38,393 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-15 02:53:38,393 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681527213260.c44270c3412c22e2c805f466aae6ff56. 2023-04-15 02:53:38,393 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681527213260.c44270c3412c22e2c805f466aae6ff56. started... 2023-04-15 02:53:38,393 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681527213260.c44270c3412c22e2c805f466aae6ff56. as already flushing 2023-04-15 02:53:38,393 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681527213260.c44270c3412c22e2c805f466aae6ff56. 2023-04-15 02:53:38,394 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-15 02:53:38,394 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-15 02:53:38,394 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:38,394 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-15 02:53:38,394 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34681,1681527209133' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-15 02:53:38,395 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:38,395 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:53:38,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:38,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:53:38,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:53:38,396 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:53:38,396 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-15 02:53:38,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:53:38,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:53:38,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:53:38,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:38,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:53:38,398 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34681,1681527209133' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-15 02:53:38,398 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@a5ecfbb[Count = 0] remaining members to acquire global barrier 2023-04-15 02:53:38,398 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-15 02:53:38,398 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:53:38,415 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:53:38,415 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:53:38,415 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:38,415 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-15 02:53:38,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:53:38,416 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-15 02:53:38,416 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-15 02:53:38,416 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34681,1681527209133' in zk 2023-04-15 02:53:38,419 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:38,419 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-15 02:53:38,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:38,419 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-15 02:53:38,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:53:38,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:53:38,419 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-15 02:53:38,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:53:38,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:53:38,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:53:38,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:38,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:53:38,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:53:38,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:38,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34681,1681527209133': 2023-04-15 02:53:38,423 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34681,1681527209133' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-15 02:53:38,423 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-15 02:53:38,423 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-15 02:53:38,423 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-15 02:53:38,423 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-15 02:53:38,423 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-15 02:53:38,427 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:53:38,427 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:53:38,427 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:53:38,427 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-15 02:53:38,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:53:38,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:53:38,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:53:38,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:53:38,428 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-15 02:53:38,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-15 02:53:38,428 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:38,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:53:38,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:53:38,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:53:38,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:53:38,429 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:38,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:53:38,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:38,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:53:38,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:53:38,432 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-15 02:53:38,432 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:38,432 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-15 02:53:38,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-15 02:53:38,432 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:53:38,432 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:53:38,432 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-15 02:53:38,432 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-15 02:53:38,433 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-15 02:53:38,433 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-15 02:53:38,432 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:38,433 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:53:38,434 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:53:38,434 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:53:38,434 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-15 02:53:38,434 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-15 02:53:38,434 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-15 02:53:38,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-15 02:53:38,435 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-15 02:53:38,463 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c44270c3412c22e2c805f466aae6ff56, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:53:38,464 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34681] ipc.CallRunner(144): callId: 53 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:38542 deadline: 1681527278462, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c44270c3412c22e2c805f466aae6ff56, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:38,467 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c44270c3412c22e2c805f466aae6ff56, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:53:38,467 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34681] ipc.CallRunner(144): callId: 52 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:38520 deadline: 1681527278465, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c44270c3412c22e2c805f466aae6ff56, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:38,468 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c44270c3412c22e2c805f466aae6ff56, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:53:38,468 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34681] ipc.CallRunner(144): callId: 54 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:38530 deadline: 1681527278467, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c44270c3412c22e2c805f466aae6ff56, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:38,470 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c44270c3412c22e2c805f466aae6ff56, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:53:38,471 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34681] ipc.CallRunner(144): callId: 53 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:38504 deadline: 1681527278469, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c44270c3412c22e2c805f466aae6ff56, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:38,527 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c44270c3412c22e2c805f466aae6ff56, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:53:38,527 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34681] ipc.CallRunner(144): callId: 43 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:38540 deadline: 1681527278523, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c44270c3412c22e2c805f466aae6ff56, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:38,534 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-15 02:53:38,535 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-15 02:53:38,537 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-15 02:53:38,538 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-15 02:53:38,538 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-15 02:53:38,539 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-15 02:53:38,539 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-15 02:53:38,539 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-15 02:53:38,539 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-15 02:53:38,540 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:53:38,540 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:53:38,541 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-15 02:53:38,541 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:38,541 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-15 02:53:38,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-15 02:53:38,541 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:38,541 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-15 02:53:38,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:53:38,541 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:53:38,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-15 02:53:38,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:53:38,542 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-15 02:53:38,542 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-15 02:53:38,542 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-15 02:53:38,542 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-15 02:53:38,543 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-15 02:53:38,543 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-15 02:53:38,543 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-15 02:53:38,544 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681527213260.c44270c3412c22e2c805f466aae6ff56. 2023-04-15 02:53:38,544 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681527213260.c44270c3412c22e2c805f466aae6ff56. started... 2023-04-15 02:53:38,544 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681527213260.c44270c3412c22e2c805f466aae6ff56. as already flushing 2023-04-15 02:53:38,544 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681527213260.c44270c3412c22e2c805f466aae6ff56. 2023-04-15 02:53:38,544 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-15 02:53:38,544 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-15 02:53:38,544 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:38,544 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-15 02:53:38,544 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34681,1681527209133' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-15 02:53:38,548 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:38,548 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:53:38,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:38,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:53:38,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:53:38,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:53:38,549 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:53:38,549 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-15 02:53:38,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:53:38,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:53:38,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:38,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:53:38,552 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34681,1681527209133' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-15 02:53:38,552 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4f714eae[Count = 0] remaining members to acquire global barrier 2023-04-15 02:53:38,552 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-15 02:53:38,552 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:53:38,557 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:53:38,557 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:53:38,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:53:38,557 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-15 02:53:38,557 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-15 02:53:38,557 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34681,1681527209133' in zk 2023-04-15 02:53:38,557 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:38,557 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-15 02:53:38,558 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:38,558 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-15 02:53:38,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:38,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:53:38,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:53:38,558 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-15 02:53:38,558 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-15 02:53:38,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:53:38,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:53:38,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:53:38,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:38,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:53:38,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:53:38,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:38,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34681,1681527209133': 2023-04-15 02:53:38,566 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34681,1681527209133' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-15 02:53:38,566 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-15 02:53:38,566 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-15 02:53:38,566 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-15 02:53:38,566 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-15 02:53:38,566 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-15 02:53:38,573 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:53:38,573 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:53:38,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:53:38,574 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-15 02:53:38,574 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:53:38,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:53:38,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:53:38,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:53:38,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:53:38,574 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-15 02:53:38,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-15 02:53:38,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:53:38,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:53:38,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:53:38,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:53:38,575 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:38,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:38,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:53:38,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:53:38,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:38,579 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:38,583 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-15 02:53:38,583 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-15 02:53:38,583 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-15 02:53:38,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-15 02:53:38,583 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:38,583 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:53:38,583 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:53:38,583 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:38,583 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-15 02:53:38,583 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:53:38,583 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-15 02:53:38,587 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-15 02:53:38,583 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:53:38,587 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:53:38,587 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-15 02:53:38,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-15 02:53:38,587 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-15 02:53:38,587 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-15 02:53:38,667 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c44270c3412c22e2c805f466aae6ff56, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:53:38,667 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34681] ipc.CallRunner(144): callId: 55 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:38542 deadline: 1681527278666, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c44270c3412c22e2c805f466aae6ff56, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:38,671 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c44270c3412c22e2c805f466aae6ff56, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:53:38,671 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34681] ipc.CallRunner(144): callId: 54 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:38520 deadline: 1681527278669, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c44270c3412c22e2c805f466aae6ff56, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:38,673 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c44270c3412c22e2c805f466aae6ff56, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:53:38,673 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34681] ipc.CallRunner(144): callId: 56 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:38530 deadline: 1681527278672, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c44270c3412c22e2c805f466aae6ff56, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:38,675 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c44270c3412c22e2c805f466aae6ff56, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:53:38,675 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34681] ipc.CallRunner(144): callId: 55 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:38504 deadline: 1681527278673, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c44270c3412c22e2c805f466aae6ff56, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:38,687 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-15 02:53:38,688 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-15 02:53:38,691 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-15 02:53:38,691 DEBUG [RS:0;jenkins-hbase20:34681-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/c44270c3412c22e2c805f466aae6ff56/.tmp/B/2b7f5f8d4c2641089789c42009ef4cef as hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/c44270c3412c22e2c805f466aae6ff56/B/2b7f5f8d4c2641089789c42009ef4cef 2023-04-15 02:53:38,693 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-15 02:53:38,693 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-15 02:53:38,693 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-15 02:53:38,694 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-15 02:53:38,695 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-15 02:53:38,695 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-15 02:53:38,695 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:53:38,695 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:53:38,697 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-15 02:53:38,697 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-15 02:53:38,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-15 02:53:38,697 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:38,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:53:38,698 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:38,698 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-15 02:53:38,698 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:53:38,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-15 02:53:38,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:53:38,699 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-15 02:53:38,704 INFO [RS:0;jenkins-hbase20:34681-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in c44270c3412c22e2c805f466aae6ff56/B of c44270c3412c22e2c805f466aae6ff56 into 2b7f5f8d4c2641089789c42009ef4cef(size=11.8 K), total size for store is 11.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-15 02:53:38,705 DEBUG [RS:0;jenkins-hbase20:34681-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for c44270c3412c22e2c805f466aae6ff56: 2023-04-15 02:53:38,705 INFO [RS:0;jenkins-hbase20:34681-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681527213260.c44270c3412c22e2c805f466aae6ff56., storeName=c44270c3412c22e2c805f466aae6ff56/B, priority=13, startTime=1681527218218; duration=0sec 2023-04-15 02:53:38,705 DEBUG [RS:0;jenkins-hbase20:34681-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-15 02:53:38,705 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-15 02:53:38,708 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-15 02:53:38,708 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-15 02:53:38,708 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-15 02:53:38,709 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-15 02:53:38,709 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-15 02:53:38,709 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-15 02:53:38,709 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681527213260.c44270c3412c22e2c805f466aae6ff56. 2023-04-15 02:53:38,709 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681527213260.c44270c3412c22e2c805f466aae6ff56. started... 2023-04-15 02:53:38,709 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681527213260.c44270c3412c22e2c805f466aae6ff56. as already flushing 2023-04-15 02:53:38,710 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681527213260.c44270c3412c22e2c805f466aae6ff56. 2023-04-15 02:53:38,710 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-15 02:53:38,710 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-15 02:53:38,710 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:38,710 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-15 02:53:38,710 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34681,1681527209133' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-15 02:53:38,711 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:38,711 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:53:38,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:38,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:53:38,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:53:38,711 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:53:38,712 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-15 02:53:38,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:53:38,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:53:38,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:53:38,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:38,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:53:38,713 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34681,1681527209133' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-15 02:53:38,713 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6fa4c9b3[Count = 0] remaining members to acquire global barrier 2023-04-15 02:53:38,713 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-15 02:53:38,713 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:53:38,714 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:53:38,714 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:53:38,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:53:38,714 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-15 02:53:38,714 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-15 02:53:38,714 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34681,1681527209133' in zk 2023-04-15 02:53:38,714 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:38,714 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-15 02:53:38,715 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:38,715 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-15 02:53:38,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:38,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:53:38,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:53:38,715 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-15 02:53:38,716 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-15 02:53:38,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:53:38,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:53:38,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:53:38,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:38,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:53:38,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:53:38,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:38,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34681,1681527209133': 2023-04-15 02:53:38,720 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34681,1681527209133' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-15 02:53:38,720 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-15 02:53:38,720 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-15 02:53:38,720 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-15 02:53:38,720 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-15 02:53:38,720 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-15 02:53:38,727 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:53:38,727 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:53:38,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:53:38,727 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:53:38,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:53:38,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:53:38,727 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-15 02:53:38,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:53:38,728 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-15 02:53:38,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-15 02:53:38,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:53:38,728 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:38,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:53:38,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:53:38,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:53:38,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:53:38,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:38,731 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:38,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:53:38,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:53:38,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:38,736 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-15 02:53:38,736 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-15 02:53:38,736 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-15 02:53:38,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-15 02:53:38,736 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-15 02:53:38,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-15 02:53:38,736 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:38,736 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:53:38,736 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:53:38,737 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:38,737 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:53:38,737 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-15 02:53:38,737 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-15 02:53:38,737 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:53:38,737 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:53:38,737 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-15 02:53:38,737 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-15 02:53:38,737 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-15 02:53:38,805 DEBUG [RS:0;jenkins-hbase20:34681-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/c44270c3412c22e2c805f466aae6ff56/.tmp/C/f5e6234a44594733a559527c3e6a1f63 as hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/c44270c3412c22e2c805f466aae6ff56/C/f5e6234a44594733a559527c3e6a1f63 2023-04-15 02:53:38,812 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=148 (bloomFilter=true), to=hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/c44270c3412c22e2c805f466aae6ff56/.tmp/B/4189f9dcfd404ba88dba3d8442bbb7cb 2023-04-15 02:53:38,820 INFO [RS:0;jenkins-hbase20:34681-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in c44270c3412c22e2c805f466aae6ff56/C of c44270c3412c22e2c805f466aae6ff56 into f5e6234a44594733a559527c3e6a1f63(size=11.8 K), total size for store is 11.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-15 02:53:38,821 DEBUG [RS:0;jenkins-hbase20:34681-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for c44270c3412c22e2c805f466aae6ff56: 2023-04-15 02:53:38,821 INFO [RS:0;jenkins-hbase20:34681-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681527213260.c44270c3412c22e2c805f466aae6ff56., storeName=c44270c3412c22e2c805f466aae6ff56/C, priority=13, startTime=1681527218218; duration=0sec 2023-04-15 02:53:38,822 DEBUG [RS:0;jenkins-hbase20:34681-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-15 02:53:38,837 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-15 02:53:38,838 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-15 02:53:38,842 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-15 02:53:38,849 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-15 02:53:38,849 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-15 02:53:38,850 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-15 02:53:38,850 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-15 02:53:38,853 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-15 02:53:38,853 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-15 02:53:38,854 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:53:38,854 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:53:38,854 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:38,854 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-15 02:53:38,855 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-15 02:53:38,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-15 02:53:38,855 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:38,855 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-15 02:53:38,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:53:38,856 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:53:38,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-15 02:53:38,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:53:38,858 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-15 02:53:38,867 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-15 02:53:38,867 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-15 02:53:38,867 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-15 02:53:38,867 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-15 02:53:38,868 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-15 02:53:38,868 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-15 02:53:38,868 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-15 02:53:38,869 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681527213260.c44270c3412c22e2c805f466aae6ff56. 2023-04-15 02:53:38,869 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681527213260.c44270c3412c22e2c805f466aae6ff56. started... 2023-04-15 02:53:38,869 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681527213260.c44270c3412c22e2c805f466aae6ff56. as already flushing 2023-04-15 02:53:38,869 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681527213260.c44270c3412c22e2c805f466aae6ff56. 2023-04-15 02:53:38,870 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-15 02:53:38,870 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=148 (bloomFilter=true), to=hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/c44270c3412c22e2c805f466aae6ff56/.tmp/C/1adba151c2d2465e8f2ae5359aad1ba5 2023-04-15 02:53:38,870 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-15 02:53:38,870 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:38,870 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-15 02:53:38,870 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34681,1681527209133' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-15 02:53:38,871 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:53:38,871 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:38,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:38,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:53:38,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:53:38,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:53:38,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:53:38,873 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:53:38,873 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-15 02:53:38,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:53:38,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:38,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:53:38,875 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34681,1681527209133' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-15 02:53:38,875 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2aa89d9[Count = 0] remaining members to acquire global barrier 2023-04-15 02:53:38,875 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-15 02:53:38,875 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:53:38,877 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:53:38,877 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:53:38,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:53:38,878 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-15 02:53:38,878 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-15 02:53:38,878 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34681,1681527209133' in zk 2023-04-15 02:53:38,879 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-15 02:53:38,879 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/c44270c3412c22e2c805f466aae6ff56/.tmp/A/94a6ea33c5964300bba31cc9d9175e53 as hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/c44270c3412c22e2c805f466aae6ff56/A/94a6ea33c5964300bba31cc9d9175e53 2023-04-15 02:53:38,879 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-15 02:53:38,879 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-15 02:53:38,881 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:38,881 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:38,882 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-15 02:53:38,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:38,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:53:38,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:53:38,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:53:38,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:53:38,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:53:38,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:38,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:53:38,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:53:38,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:38,890 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/c44270c3412c22e2c805f466aae6ff56/A/94a6ea33c5964300bba31cc9d9175e53, entries=200, sequenceid=148, filesize=14.1 K 2023-04-15 02:53:38,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34681,1681527209133': 2023-04-15 02:53:38,891 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34681,1681527209133' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-15 02:53:38,891 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-15 02:53:38,891 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-15 02:53:38,891 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-15 02:53:38,891 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-15 02:53:38,891 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-15 02:53:38,905 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/c44270c3412c22e2c805f466aae6ff56/.tmp/B/4189f9dcfd404ba88dba3d8442bbb7cb as hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/c44270c3412c22e2c805f466aae6ff56/B/4189f9dcfd404ba88dba3d8442bbb7cb 2023-04-15 02:53:38,905 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:53:38,905 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:53:38,906 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-15 02:53:38,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:53:38,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:53:38,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:53:38,906 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:53:38,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:53:38,906 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-15 02:53:38,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-15 02:53:38,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:53:38,907 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:38,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:53:38,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:53:38,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:53:38,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:53:38,909 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:38,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:38,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:53:38,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:53:38,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:38,917 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-15 02:53:38,918 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:38,918 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-15 02:53:38,918 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:53:38,918 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-15 02:53:38,918 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-15 02:53:38,918 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-15 02:53:38,918 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:53:38,918 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:38,918 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:53:38,918 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:53:38,918 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-15 02:53:38,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-15 02:53:38,921 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-15 02:53:38,918 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:53:38,921 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-15 02:53:38,922 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-15 02:53:38,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-15 02:53:38,922 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-15 02:53:38,929 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/c44270c3412c22e2c805f466aae6ff56/B/4189f9dcfd404ba88dba3d8442bbb7cb, entries=150, sequenceid=148, filesize=11.8 K 2023-04-15 02:53:38,932 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/c44270c3412c22e2c805f466aae6ff56/.tmp/C/1adba151c2d2465e8f2ae5359aad1ba5 as hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/c44270c3412c22e2c805f466aae6ff56/C/1adba151c2d2465e8f2ae5359aad1ba5 2023-04-15 02:53:38,945 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/c44270c3412c22e2c805f466aae6ff56/C/1adba151c2d2465e8f2ae5359aad1ba5, entries=150, sequenceid=148, filesize=11.8 K 2023-04-15 02:53:38,947 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~114.05 KB/116790, heapSize ~339.38 KB/347520, currentSize=60.38 KB/61830 for c44270c3412c22e2c805f466aae6ff56 in 631ms, sequenceid=148, compaction requested=false 2023-04-15 02:53:38,948 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for c44270c3412c22e2c805f466aae6ff56: 2023-04-15 02:53:38,971 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing c44270c3412c22e2c805f466aae6ff56 3/3 column families, dataSize=67.09 KB heapSize=199.97 KB 2023-04-15 02:53:38,972 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c44270c3412c22e2c805f466aae6ff56, store=A 2023-04-15 02:53:38,972 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-15 02:53:38,972 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c44270c3412c22e2c805f466aae6ff56, store=B 2023-04-15 02:53:38,972 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-15 02:53:38,972 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c44270c3412c22e2c805f466aae6ff56, store=C 2023-04-15 02:53:38,972 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-15 02:53:38,972 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34681] regionserver.HRegion(9158): Flush requested on c44270c3412c22e2c805f466aae6ff56 2023-04-15 02:53:39,021 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-15 02:53:39,022 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-15 02:53:39,024 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-15 02:53:39,025 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-15 02:53:39,026 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-15 02:53:39,026 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-15 02:53:39,026 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-15 02:53:39,026 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-15 02:53:39,026 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-15 02:53:39,027 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:53:39,027 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:53:39,028 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-15 02:53:39,028 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:39,028 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-15 02:53:39,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-15 02:53:39,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:53:39,029 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:39,029 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-15 02:53:39,029 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:53:39,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-15 02:53:39,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:53:39,029 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-15 02:53:39,030 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-15 02:53:39,030 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-15 02:53:39,030 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-15 02:53:39,030 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-15 02:53:39,031 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-15 02:53:39,034 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-15 02:53:39,034 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681527213260.c44270c3412c22e2c805f466aae6ff56. 2023-04-15 02:53:39,034 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681527213260.c44270c3412c22e2c805f466aae6ff56. started... 2023-04-15 02:53:39,034 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681527213260.c44270c3412c22e2c805f466aae6ff56. as already flushing 2023-04-15 02:53:39,035 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681527213260.c44270c3412c22e2c805f466aae6ff56. 2023-04-15 02:53:39,034 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-15 02:53:39,035 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-15 02:53:39,035 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-15 02:53:39,035 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:39,035 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-15 02:53:39,035 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34681,1681527209133' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-15 02:53:39,036 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:39,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:39,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:53:39,036 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:53:39,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:53:39,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:53:39,037 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:53:39,037 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-15 02:53:39,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:53:39,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:53:39,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:39,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:53:39,039 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34681,1681527209133' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-15 02:53:39,039 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7ef5bd20[Count = 0] remaining members to acquire global barrier 2023-04-15 02:53:39,039 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-15 02:53:39,039 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:53:39,040 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:53:39,041 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:53:39,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:53:39,041 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-15 02:53:39,041 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-15 02:53:39,041 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34681,1681527209133' in zk 2023-04-15 02:53:39,041 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:39,041 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-15 02:53:39,042 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-15 02:53:39,042 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:39,042 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-15 02:53:39,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:39,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:53:39,042 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-15 02:53:39,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:53:39,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:53:39,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:53:39,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:53:39,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:39,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:53:39,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:53:39,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:39,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34681,1681527209133': 2023-04-15 02:53:39,049 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34681,1681527209133' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-15 02:53:39,049 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-15 02:53:39,049 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-15 02:53:39,049 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-15 02:53:39,049 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-15 02:53:39,049 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-15 02:53:39,051 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c44270c3412c22e2c805f466aae6ff56, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:53:39,052 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34681] ipc.CallRunner(144): callId: 60 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:38504 deadline: 1681527279047, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c44270c3412c22e2c805f466aae6ff56, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:39,052 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c44270c3412c22e2c805f466aae6ff56, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:53:39,052 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34681] ipc.CallRunner(144): callId: 62 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:38542 deadline: 1681527279049, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c44270c3412c22e2c805f466aae6ff56, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:39,052 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:53:39,052 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-15 02:53:39,053 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:53:39,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:53:39,054 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-15 02:53:39,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-15 02:53:39,054 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:53:39,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:53:39,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:53:39,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:53:39,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:53:39,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:53:39,055 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:39,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:53:39,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:53:39,056 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c44270c3412c22e2c805f466aae6ff56, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:53:39,056 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34681] ipc.CallRunner(144): callId: 63 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:38530 deadline: 1681527279051, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c44270c3412c22e2c805f466aae6ff56, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:39,056 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c44270c3412c22e2c805f466aae6ff56, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:53:39,057 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34681] ipc.CallRunner(144): callId: 61 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:38520 deadline: 1681527279054, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c44270c3412c22e2c805f466aae6ff56, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:39,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:53:39,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:39,058 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:39,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:53:39,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:53:39,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:39,066 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-15 02:53:39,066 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:39,066 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-15 02:53:39,066 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:53:39,066 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:53:39,066 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:39,066 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:53:39,066 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:53:39,066 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-15 02:53:39,066 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-15 02:53:39,066 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-15 02:53:39,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-15 02:53:39,066 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:53:39,066 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-15 02:53:39,066 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-15 02:53:39,066 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-15 02:53:39,073 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-15 02:53:39,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-15 02:53:39,155 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c44270c3412c22e2c805f466aae6ff56, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:53:39,155 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34681] ipc.CallRunner(144): callId: 62 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:38504 deadline: 1681527279153, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c44270c3412c22e2c805f466aae6ff56, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:39,156 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c44270c3412c22e2c805f466aae6ff56, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:53:39,156 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34681] ipc.CallRunner(144): callId: 64 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:38542 deadline: 1681527279155, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c44270c3412c22e2c805f466aae6ff56, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:39,159 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c44270c3412c22e2c805f466aae6ff56, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:53:39,160 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34681] ipc.CallRunner(144): callId: 63 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:38520 deadline: 1681527279158, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c44270c3412c22e2c805f466aae6ff56, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:39,160 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c44270c3412c22e2c805f466aae6ff56, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:53:39,160 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34681] ipc.CallRunner(144): callId: 65 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:38530 deadline: 1681527279158, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c44270c3412c22e2c805f466aae6ff56, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:39,167 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-15 02:53:39,167 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-15 02:53:39,169 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-15 02:53:39,171 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-15 02:53:39,171 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-15 02:53:39,171 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-15 02:53:39,171 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-15 02:53:39,172 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-15 02:53:39,172 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-15 02:53:39,172 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:53:39,172 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:53:39,173 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-15 02:53:39,173 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-15 02:53:39,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-15 02:53:39,173 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:39,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:53:39,173 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:39,173 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-15 02:53:39,173 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:53:39,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-15 02:53:39,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:53:39,174 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-15 02:53:39,174 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-15 02:53:39,174 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-15 02:53:39,174 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-15 02:53:39,174 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-15 02:53:39,175 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-15 02:53:39,175 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-15 02:53:39,175 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-15 02:53:39,175 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681527213260.c44270c3412c22e2c805f466aae6ff56. 2023-04-15 02:53:39,175 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681527213260.c44270c3412c22e2c805f466aae6ff56. started... 2023-04-15 02:53:39,175 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681527213260.c44270c3412c22e2c805f466aae6ff56. as already flushing 2023-04-15 02:53:39,175 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681527213260.c44270c3412c22e2c805f466aae6ff56. 2023-04-15 02:53:39,175 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-15 02:53:39,176 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-15 02:53:39,176 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:39,176 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-15 02:53:39,176 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34681,1681527209133' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-15 02:53:39,177 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:39,177 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:53:39,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:39,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:53:39,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:53:39,177 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:53:39,177 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-15 02:53:39,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:53:39,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:53:39,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:53:39,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:39,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:53:39,179 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34681,1681527209133' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-15 02:53:39,179 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3bb4886e[Count = 0] remaining members to acquire global barrier 2023-04-15 02:53:39,179 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-15 02:53:39,179 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:53:39,180 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:53:39,180 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:53:39,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:53:39,180 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-15 02:53:39,180 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-15 02:53:39,180 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:39,180 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-15 02:53:39,180 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34681,1681527209133' in zk 2023-04-15 02:53:39,181 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:39,181 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-15 02:53:39,181 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-15 02:53:39,181 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-15 02:53:39,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:39,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:53:39,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:53:39,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:53:39,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:53:39,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:53:39,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:39,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:53:39,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:53:39,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:39,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34681,1681527209133': 2023-04-15 02:53:39,184 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34681,1681527209133' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-15 02:53:39,184 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-15 02:53:39,184 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-15 02:53:39,184 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-15 02:53:39,184 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-15 02:53:39,185 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-15 02:53:39,186 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:53:39,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:53:39,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:53:39,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:53:39,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:53:39,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:53:39,189 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:39,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:53:39,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:53:39,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:39,190 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:39,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:53:39,190 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:53:39,190 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-15 02:53:39,190 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:53:39,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:53:39,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:53:39,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:39,191 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-15 02:53:39,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-15 02:53:39,194 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:39,194 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-15 02:53:39,194 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-15 02:53:39,194 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-15 02:53:39,194 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-15 02:53:39,195 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-15 02:53:39,194 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:53:39,195 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-15 02:53:39,195 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-15 02:53:39,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-15 02:53:39,195 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:53:39,195 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-15 02:53:39,195 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:39,196 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:53:39,196 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:53:39,196 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:53:39,296 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-15 02:53:39,296 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-15 02:53:39,299 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-15 02:53:39,303 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-15 02:53:39,303 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-15 02:53:39,303 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-15 02:53:39,303 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-15 02:53:39,304 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-15 02:53:39,304 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-15 02:53:39,304 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:53:39,304 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:53:39,305 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-15 02:53:39,305 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:39,305 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-15 02:53:39,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-15 02:53:39,305 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:39,305 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-15 02:53:39,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:53:39,306 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:53:39,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-15 02:53:39,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:53:39,306 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-15 02:53:39,307 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-15 02:53:39,307 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-15 02:53:39,307 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-15 02:53:39,307 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-15 02:53:39,307 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-15 02:53:39,308 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-15 02:53:39,308 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681527213260.c44270c3412c22e2c805f466aae6ff56. 2023-04-15 02:53:39,308 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-15 02:53:39,308 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681527213260.c44270c3412c22e2c805f466aae6ff56. started... 2023-04-15 02:53:39,308 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681527213260.c44270c3412c22e2c805f466aae6ff56. as already flushing 2023-04-15 02:53:39,308 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681527213260.c44270c3412c22e2c805f466aae6ff56. 2023-04-15 02:53:39,308 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-15 02:53:39,308 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-15 02:53:39,308 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:39,308 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-15 02:53:39,308 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34681,1681527209133' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-15 02:53:39,309 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:39,309 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:53:39,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:39,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:53:39,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:53:39,310 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:53:39,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:53:39,310 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-15 02:53:39,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:53:39,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:53:39,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:39,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:53:39,311 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34681,1681527209133' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-15 02:53:39,311 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@729437fa[Count = 0] remaining members to acquire global barrier 2023-04-15 02:53:39,311 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-15 02:53:39,311 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:53:39,312 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:53:39,312 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:53:39,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:53:39,312 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-15 02:53:39,312 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-15 02:53:39,312 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34681,1681527209133' in zk 2023-04-15 02:53:39,314 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-15 02:53:39,314 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:39,314 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-15 02:53:39,314 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-15 02:53:39,315 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34681,1681527209133' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-15 02:53:39,315 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-15 02:53:39,315 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-15 02:53:39,315 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-15 02:53:39,315 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-15 02:53:39,315 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-15 02:53:39,315 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-15 02:53:39,316 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:53:39,316 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-15 02:53:39,316 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:53:39,316 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:53:39,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:53:39,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:53:39,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:53:39,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:53:39,317 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-15 02:53:39,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-15 02:53:39,317 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:39,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:53:39,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:53:39,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:53:39,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:53:39,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:53:39,320 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:39,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:39,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:53:39,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:53:39,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:39,328 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:39,328 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:53:39,328 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:53:39,328 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-15 02:53:39,328 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-15 02:53:39,328 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-15 02:53:39,328 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:39,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-15 02:53:39,328 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:53:39,328 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-15 02:53:39,328 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:53:39,328 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:53:39,328 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-15 02:53:39,328 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-15 02:53:39,329 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-15 02:53:39,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-15 02:53:39,329 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-15 02:53:39,329 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-15 02:53:39,330 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-15 02:53:39,360 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c44270c3412c22e2c805f466aae6ff56, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:53:39,360 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34681] ipc.CallRunner(144): callId: 64 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:38504 deadline: 1681527279357, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c44270c3412c22e2c805f466aae6ff56, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:39,360 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c44270c3412c22e2c805f466aae6ff56, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:53:39,360 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34681] ipc.CallRunner(144): callId: 66 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:38542 deadline: 1681527279358, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c44270c3412c22e2c805f466aae6ff56, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:39,366 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c44270c3412c22e2c805f466aae6ff56, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:53:39,367 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34681] ipc.CallRunner(144): callId: 65 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:38520 deadline: 1681527279362, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c44270c3412c22e2c805f466aae6ff56, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:39,367 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c44270c3412c22e2c805f466aae6ff56, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:53:39,367 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34681] ipc.CallRunner(144): callId: 67 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:38530 deadline: 1681527279363, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c44270c3412c22e2c805f466aae6ff56, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:39,398 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=163 (bloomFilter=true), to=hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/c44270c3412c22e2c805f466aae6ff56/.tmp/A/cc98a063c4a940448766c211027bf809 2023-04-15 02:53:39,430 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-15 02:53:39,430 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-15 02:53:39,432 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-15 02:53:39,434 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-15 02:53:39,435 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-15 02:53:39,436 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-15 02:53:39,436 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-15 02:53:39,436 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-15 02:53:39,436 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-15 02:53:39,437 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:53:39,437 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:53:39,438 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-15 02:53:39,438 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:39,438 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-15 02:53:39,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-15 02:53:39,438 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:39,438 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-15 02:53:39,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:53:39,438 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:53:39,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-15 02:53:39,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:53:39,439 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-15 02:53:39,439 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-15 02:53:39,439 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-15 02:53:39,439 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-15 02:53:39,439 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-15 02:53:39,440 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-15 02:53:39,440 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-15 02:53:39,440 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-15 02:53:39,440 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681527213260.c44270c3412c22e2c805f466aae6ff56. 2023-04-15 02:53:39,440 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681527213260.c44270c3412c22e2c805f466aae6ff56. started... 2023-04-15 02:53:39,440 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681527213260.c44270c3412c22e2c805f466aae6ff56. as already flushing 2023-04-15 02:53:39,440 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681527213260.c44270c3412c22e2c805f466aae6ff56. 2023-04-15 02:53:39,440 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-15 02:53:39,440 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-15 02:53:39,440 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:39,440 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-15 02:53:39,440 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34681,1681527209133' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-15 02:53:39,442 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:53:39,442 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:39,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:39,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:53:39,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:53:39,442 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:53:39,442 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-15 02:53:39,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:53:39,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:53:39,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:53:39,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:39,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:53:39,444 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34681,1681527209133' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-15 02:53:39,444 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@24268c30[Count = 0] remaining members to acquire global barrier 2023-04-15 02:53:39,444 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-15 02:53:39,444 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:53:39,445 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:53:39,445 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:53:39,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:53:39,445 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-15 02:53:39,445 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-15 02:53:39,445 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34681,1681527209133' in zk 2023-04-15 02:53:39,445 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:39,445 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-15 02:53:39,446 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:39,446 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-15 02:53:39,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:39,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:53:39,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:53:39,446 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-15 02:53:39,446 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-15 02:53:39,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:53:39,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:53:39,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:53:39,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:39,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:53:39,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:53:39,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:39,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34681,1681527209133': 2023-04-15 02:53:39,450 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34681,1681527209133' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-15 02:53:39,450 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-15 02:53:39,450 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-15 02:53:39,450 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-15 02:53:39,450 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-15 02:53:39,450 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-15 02:53:39,457 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:53:39,457 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:53:39,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:53:39,457 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:53:39,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:53:39,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:53:39,457 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-15 02:53:39,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:53:39,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:53:39,457 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:39,457 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-15 02:53:39,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-15 02:53:39,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:53:39,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:53:39,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:53:39,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:53:39,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:39,459 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:39,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:53:39,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:53:39,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:39,466 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-15 02:53:39,466 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-15 02:53:39,466 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-15 02:53:39,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-15 02:53:39,466 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-15 02:53:39,471 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-15 02:53:39,471 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-15 02:53:39,471 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:39,471 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:53:39,471 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:53:39,471 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:39,471 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-15 02:53:39,471 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-15 02:53:39,471 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:53:39,472 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:53:39,472 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:53:39,472 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-15 02:53:39,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-15 02:53:39,536 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c44270c3412c22e2c805f466aae6ff56, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:53:39,536 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34681] ipc.CallRunner(144): callId: 45 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:38540 deadline: 1681527279534, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c44270c3412c22e2c805f466aae6ff56, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:39,572 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-15 02:53:39,572 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-15 02:53:39,574 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-15 02:53:39,575 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-15 02:53:39,576 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-15 02:53:39,576 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-15 02:53:39,576 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-15 02:53:39,577 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-15 02:53:39,577 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-15 02:53:39,577 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:53:39,577 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:53:39,578 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-15 02:53:39,578 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:39,578 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-15 02:53:39,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-15 02:53:39,578 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:39,578 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-15 02:53:39,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:53:39,579 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:53:39,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-15 02:53:39,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:53:39,579 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-15 02:53:39,579 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-15 02:53:39,579 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-15 02:53:39,579 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-15 02:53:39,579 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-15 02:53:39,581 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-15 02:53:39,582 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-15 02:53:39,582 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-15 02:53:39,582 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681527213260.c44270c3412c22e2c805f466aae6ff56. 2023-04-15 02:53:39,582 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681527213260.c44270c3412c22e2c805f466aae6ff56. started... 2023-04-15 02:53:39,582 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681527213260.c44270c3412c22e2c805f466aae6ff56. as already flushing 2023-04-15 02:53:39,582 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681527213260.c44270c3412c22e2c805f466aae6ff56. 2023-04-15 02:53:39,582 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-15 02:53:39,582 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-15 02:53:39,582 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:39,582 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-15 02:53:39,582 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34681,1681527209133' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-15 02:53:39,590 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:39,590 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:53:39,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:39,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:53:39,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:53:39,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:53:39,591 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:53:39,591 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-15 02:53:39,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:53:39,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:53:39,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:39,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:53:39,593 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34681,1681527209133' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-15 02:53:39,593 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@77e93929[Count = 0] remaining members to acquire global barrier 2023-04-15 02:53:39,593 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-15 02:53:39,593 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:53:39,594 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:53:39,594 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:53:39,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:53:39,594 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-15 02:53:39,594 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-15 02:53:39,594 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:39,594 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-15 02:53:39,594 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34681,1681527209133' in zk 2023-04-15 02:53:39,595 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-15 02:53:39,595 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:39,595 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-15 02:53:39,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:39,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:53:39,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:53:39,595 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-15 02:53:39,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:53:39,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:53:39,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:53:39,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:39,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:53:39,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:53:39,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:39,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34681,1681527209133': 2023-04-15 02:53:39,600 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34681,1681527209133' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-15 02:53:39,600 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-15 02:53:39,600 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-15 02:53:39,600 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-15 02:53:39,600 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-15 02:53:39,600 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-15 02:53:39,607 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:53:39,607 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:53:39,607 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:53:39,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:53:39,607 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-15 02:53:39,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:53:39,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:53:39,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:53:39,607 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-15 02:53:39,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-15 02:53:39,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:53:39,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:53:39,608 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:39,609 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:39,611 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:39,611 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:53:39,611 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:53:39,611 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:39,611 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:53:39,611 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:53:39,611 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:53:39,612 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-15 02:53:39,612 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-15 02:53:39,612 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-15 02:53:39,612 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-15 02:53:39,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:53:39,612 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-15 02:53:39,615 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-15 02:53:39,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-15 02:53:39,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:53:39,615 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-15 02:53:39,615 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-15 02:53:39,615 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-15 02:53:39,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-15 02:53:39,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:53:39,616 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-15 02:53:39,665 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c44270c3412c22e2c805f466aae6ff56, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:53:39,665 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34681] ipc.CallRunner(144): callId: 66 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:38504 deadline: 1681527279663, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c44270c3412c22e2c805f466aae6ff56, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:39,666 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c44270c3412c22e2c805f466aae6ff56, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:53:39,666 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34681] ipc.CallRunner(144): callId: 68 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:38542 deadline: 1681527279664, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c44270c3412c22e2c805f466aae6ff56, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:39,669 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c44270c3412c22e2c805f466aae6ff56, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:53:39,669 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34681] ipc.CallRunner(144): callId: 67 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:38520 deadline: 1681527279668, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c44270c3412c22e2c805f466aae6ff56, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:39,672 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c44270c3412c22e2c805f466aae6ff56, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:53:39,672 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34681] ipc.CallRunner(144): callId: 69 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:38530 deadline: 1681527279672, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c44270c3412c22e2c805f466aae6ff56, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:39,715 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-15 02:53:39,716 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-15 02:53:39,718 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-15 02:53:39,720 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-15 02:53:39,720 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-15 02:53:39,720 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-15 02:53:39,720 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-15 02:53:39,720 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-15 02:53:39,720 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-15 02:53:39,721 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:53:39,721 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:53:39,722 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:39,722 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-15 02:53:39,722 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-15 02:53:39,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-15 02:53:39,722 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:39,722 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-15 02:53:39,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:53:39,722 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:53:39,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-15 02:53:39,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:53:39,723 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-15 02:53:39,723 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-15 02:53:39,723 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-15 02:53:39,723 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-15 02:53:39,723 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-15 02:53:39,723 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-15 02:53:39,723 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-15 02:53:39,723 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-15 02:53:39,723 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681527213260.c44270c3412c22e2c805f466aae6ff56. 2023-04-15 02:53:39,723 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681527213260.c44270c3412c22e2c805f466aae6ff56. started... 2023-04-15 02:53:39,723 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681527213260.c44270c3412c22e2c805f466aae6ff56. as already flushing 2023-04-15 02:53:39,723 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681527213260.c44270c3412c22e2c805f466aae6ff56. 2023-04-15 02:53:39,723 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-15 02:53:39,723 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-15 02:53:39,723 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:39,723 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-15 02:53:39,723 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34681,1681527209133' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-15 02:53:39,724 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:39,724 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:53:39,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:39,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:53:39,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:53:39,725 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:53:39,725 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-15 02:53:39,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:53:39,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:53:39,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:53:39,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:39,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:53:39,726 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34681,1681527209133' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-15 02:53:39,726 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-15 02:53:39,726 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@68813842[Count = 0] remaining members to acquire global barrier 2023-04-15 02:53:39,726 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:53:39,727 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:53:39,727 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:53:39,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:53:39,727 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-15 02:53:39,727 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-15 02:53:39,727 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34681,1681527209133' in zk 2023-04-15 02:53:39,727 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:39,727 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-15 02:53:39,728 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-15 02:53:39,728 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-15 02:53:39,728 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-15 02:53:39,728 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:39,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:39,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:53:39,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:53:39,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:53:39,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:53:39,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:53:39,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:39,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:53:39,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:53:39,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:39,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34681,1681527209133': 2023-04-15 02:53:39,731 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34681,1681527209133' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-15 02:53:39,731 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-15 02:53:39,731 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-15 02:53:39,731 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-15 02:53:39,731 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-15 02:53:39,731 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-15 02:53:39,732 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:53:39,732 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:53:39,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:53:39,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:53:39,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:53:39,732 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:53:39,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:53:39,732 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-15 02:53:39,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:53:39,733 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-15 02:53:39,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-15 02:53:39,733 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:39,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:53:39,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:53:39,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:53:39,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:53:39,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:39,734 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:39,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:53:39,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:53:39,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:39,737 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:39,737 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-15 02:53:39,737 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:53:39,737 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-15 02:53:39,737 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-15 02:53:39,737 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:53:39,737 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-15 02:53:39,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-15 02:53:39,737 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-15 02:53:39,737 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-15 02:53:39,737 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:39,737 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:53:39,737 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:53:39,737 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:53:39,737 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-15 02:53:39,737 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-15 02:53:39,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-15 02:53:39,738 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-15 02:53:39,818 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=163 (bloomFilter=true), to=hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/c44270c3412c22e2c805f466aae6ff56/.tmp/B/5e3a6ea18fae44eaa9eeb08894d78035 2023-04-15 02:53:39,838 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-15 02:53:39,838 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-15 02:53:39,841 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-15 02:53:39,843 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-15 02:53:39,843 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-15 02:53:39,843 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-15 02:53:39,843 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-15 02:53:39,843 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-15 02:53:39,843 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-15 02:53:39,844 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:53:39,844 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:53:39,845 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:39,845 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-15 02:53:39,845 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-15 02:53:39,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-15 02:53:39,846 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:39,846 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-15 02:53:39,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:53:39,850 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:53:39,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-15 02:53:39,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:53:39,851 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-15 02:53:39,855 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-15 02:53:39,855 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-15 02:53:39,855 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-15 02:53:39,855 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-15 02:53:39,855 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-15 02:53:39,856 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-15 02:53:39,856 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-15 02:53:39,856 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681527213260.c44270c3412c22e2c805f466aae6ff56. 2023-04-15 02:53:39,856 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681527213260.c44270c3412c22e2c805f466aae6ff56. started... 2023-04-15 02:53:39,856 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681527213260.c44270c3412c22e2c805f466aae6ff56. as already flushing 2023-04-15 02:53:39,856 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681527213260.c44270c3412c22e2c805f466aae6ff56. 2023-04-15 02:53:39,856 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-15 02:53:39,856 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-15 02:53:39,856 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:39,856 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-15 02:53:39,856 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34681,1681527209133' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-15 02:53:39,869 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:39,869 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:53:39,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:39,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:53:39,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:53:39,869 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:53:39,869 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-15 02:53:39,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:53:39,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:53:39,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:53:39,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:39,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:53:39,871 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34681,1681527209133' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-15 02:53:39,871 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3bf9e94f[Count = 0] remaining members to acquire global barrier 2023-04-15 02:53:39,871 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-15 02:53:39,871 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:53:39,873 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:53:39,873 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:53:39,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:53:39,874 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-15 02:53:39,874 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-15 02:53:39,874 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34681,1681527209133' in zk 2023-04-15 02:53:39,874 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:39,874 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-15 02:53:39,875 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-15 02:53:39,875 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:39,875 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-15 02:53:39,875 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-15 02:53:39,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:39,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:53:39,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:53:39,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:53:39,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:53:39,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:53:39,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:39,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:53:39,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:53:39,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:39,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34681,1681527209133': 2023-04-15 02:53:39,878 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34681,1681527209133' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-15 02:53:39,878 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-15 02:53:39,878 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-15 02:53:39,878 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-15 02:53:39,878 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-15 02:53:39,879 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-15 02:53:39,880 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:53:39,880 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-15 02:53:39,880 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:53:39,880 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:53:39,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:53:39,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:53:39,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:53:39,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:53:39,880 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:39,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:53:39,881 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-15 02:53:39,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-15 02:53:39,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:53:39,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:53:39,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:53:39,882 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:39,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:53:39,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:39,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:53:39,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:53:39,884 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:39,884 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:53:39,884 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:53:39,885 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:39,885 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:53:39,885 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:53:39,885 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:53:39,885 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-15 02:53:39,885 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-15 02:53:39,885 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-15 02:53:39,885 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-15 02:53:39,885 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-15 02:53:39,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-15 02:53:39,885 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-15 02:53:39,885 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-15 02:53:39,885 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-15 02:53:39,885 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-15 02:53:39,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-15 02:53:39,886 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-15 02:53:39,985 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-15 02:53:39,986 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-15 02:53:39,989 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-15 02:53:39,994 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-15 02:53:39,994 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-15 02:53:39,994 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-15 02:53:39,994 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-15 02:53:39,994 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-15 02:53:39,994 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-15 02:53:39,995 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:53:39,995 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:53:39,995 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-15 02:53:39,996 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-15 02:53:39,996 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:39,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-15 02:53:39,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:53:39,996 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:39,996 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-15 02:53:39,996 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:53:39,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-15 02:53:39,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:53:39,997 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-15 02:53:39,998 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-15 02:53:39,998 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-15 02:53:39,999 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-15 02:53:39,999 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-15 02:53:39,999 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-15 02:53:40,008 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-15 02:53:40,008 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-15 02:53:40,008 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681527213260.c44270c3412c22e2c805f466aae6ff56. 2023-04-15 02:53:40,008 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681527213260.c44270c3412c22e2c805f466aae6ff56. started... 2023-04-15 02:53:40,008 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681527213260.c44270c3412c22e2c805f466aae6ff56. as already flushing 2023-04-15 02:53:40,008 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681527213260.c44270c3412c22e2c805f466aae6ff56. 2023-04-15 02:53:40,008 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-15 02:53:40,008 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-15 02:53:40,008 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:40,009 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-15 02:53:40,009 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34681,1681527209133' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-15 02:53:40,010 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:40,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:40,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:53:40,011 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:53:40,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:53:40,011 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:53:40,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:53:40,011 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-15 02:53:40,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:53:40,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:53:40,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:40,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:53:40,012 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34681,1681527209133' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-15 02:53:40,012 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2b2eb50c[Count = 0] remaining members to acquire global barrier 2023-04-15 02:53:40,012 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-15 02:53:40,012 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:53:40,013 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:53:40,013 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:53:40,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:53:40,013 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-15 02:53:40,013 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-15 02:53:40,013 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34681,1681527209133' in zk 2023-04-15 02:53:40,013 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:40,014 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-15 02:53:40,014 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:40,014 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-15 02:53:40,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:40,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:53:40,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:53:40,014 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-15 02:53:40,014 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-15 02:53:40,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:53:40,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:53:40,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:53:40,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:40,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:53:40,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:53:40,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:40,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34681,1681527209133': 2023-04-15 02:53:40,017 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34681,1681527209133' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-15 02:53:40,017 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-15 02:53:40,017 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-15 02:53:40,017 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-15 02:53:40,017 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-15 02:53:40,017 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-15 02:53:40,018 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:53:40,018 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-15 02:53:40,018 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:53:40,018 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:53:40,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:53:40,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:53:40,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:53:40,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:53:40,019 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-15 02:53:40,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-15 02:53:40,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:53:40,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:53:40,020 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:40,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:53:40,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:53:40,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:53:40,021 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:40,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:40,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:53:40,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:53:40,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:40,025 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:40,025 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-15 02:53:40,025 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-15 02:53:40,025 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:53:40,025 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-15 02:53:40,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-15 02:53:40,025 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:53:40,025 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:40,025 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:53:40,025 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:53:40,025 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:53:40,025 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-15 02:53:40,025 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-15 02:53:40,025 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-15 02:53:40,025 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-15 02:53:40,025 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-15 02:53:40,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-15 02:53:40,026 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-15 02:53:40,026 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-15 02:53:40,126 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-15 02:53:40,126 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-15 02:53:40,128 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-15 02:53:40,129 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-15 02:53:40,130 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-15 02:53:40,130 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-15 02:53:40,130 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-15 02:53:40,130 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-15 02:53:40,130 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-15 02:53:40,130 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:53:40,130 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:53:40,131 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:40,131 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-15 02:53:40,131 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-15 02:53:40,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-15 02:53:40,131 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:40,131 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-15 02:53:40,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:53:40,132 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:53:40,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-15 02:53:40,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:53:40,132 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-15 02:53:40,132 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-15 02:53:40,132 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-15 02:53:40,132 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-15 02:53:40,132 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-15 02:53:40,132 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-15 02:53:40,133 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-15 02:53:40,133 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-15 02:53:40,133 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681527213260.c44270c3412c22e2c805f466aae6ff56. 2023-04-15 02:53:40,133 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681527213260.c44270c3412c22e2c805f466aae6ff56. started... 2023-04-15 02:53:40,133 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681527213260.c44270c3412c22e2c805f466aae6ff56. as already flushing 2023-04-15 02:53:40,133 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681527213260.c44270c3412c22e2c805f466aae6ff56. 2023-04-15 02:53:40,133 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-15 02:53:40,133 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-15 02:53:40,133 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:40,133 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-15 02:53:40,133 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34681,1681527209133' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-15 02:53:40,134 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:40,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:40,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:53:40,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:53:40,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:53:40,134 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:53:40,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:53:40,135 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:53:40,135 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-15 02:53:40,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:53:40,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:40,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:53:40,136 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34681,1681527209133' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-15 02:53:40,136 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1530e131[Count = 0] remaining members to acquire global barrier 2023-04-15 02:53:40,136 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-15 02:53:40,136 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:53:40,136 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:53:40,136 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:53:40,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:53:40,136 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-15 02:53:40,137 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:40,137 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-15 02:53:40,137 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-15 02:53:40,137 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34681,1681527209133' in zk 2023-04-15 02:53:40,137 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:40,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:40,137 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-15 02:53:40,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:53:40,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:53:40,137 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-15 02:53:40,138 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-15 02:53:40,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:53:40,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:53:40,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:53:40,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:40,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:53:40,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:53:40,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:40,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34681,1681527209133': 2023-04-15 02:53:40,139 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34681,1681527209133' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-15 02:53:40,139 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-15 02:53:40,139 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-15 02:53:40,139 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-15 02:53:40,139 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-15 02:53:40,139 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-15 02:53:40,140 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:53:40,140 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:53:40,140 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:53:40,140 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-15 02:53:40,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:53:40,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:53:40,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:53:40,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:53:40,141 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:40,141 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-15 02:53:40,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:53:40,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-15 02:53:40,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:53:40,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:53:40,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:53:40,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:53:40,142 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:40,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:40,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:53:40,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:53:40,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:40,153 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:40,153 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-15 02:53:40,153 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:53:40,153 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:53:40,153 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:40,153 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:53:40,153 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:53:40,153 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:53:40,153 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-15 02:53:40,153 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-15 02:53:40,153 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-15 02:53:40,154 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-15 02:53:40,153 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-15 02:53:40,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-15 02:53:40,154 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-15 02:53:40,154 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-15 02:53:40,154 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-15 02:53:40,154 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-15 02:53:40,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-15 02:53:40,171 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c44270c3412c22e2c805f466aae6ff56, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:53:40,171 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34681] ipc.CallRunner(144): callId: 69 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:38520 deadline: 1681527280170, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c44270c3412c22e2c805f466aae6ff56, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:40,171 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c44270c3412c22e2c805f466aae6ff56, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:53:40,171 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34681] ipc.CallRunner(144): callId: 68 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:38504 deadline: 1681527280171, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c44270c3412c22e2c805f466aae6ff56, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:40,174 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c44270c3412c22e2c805f466aae6ff56, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:53:40,175 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34681] ipc.CallRunner(144): callId: 70 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:38542 deadline: 1681527280174, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c44270c3412c22e2c805f466aae6ff56, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:40,176 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c44270c3412c22e2c805f466aae6ff56, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:53:40,176 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34681] ipc.CallRunner(144): callId: 71 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:38530 deadline: 1681527280175, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c44270c3412c22e2c805f466aae6ff56, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:40,254 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-15 02:53:40,255 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-15 02:53:40,257 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=163 (bloomFilter=true), to=hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/c44270c3412c22e2c805f466aae6ff56/.tmp/C/4f01c5f98a3d4c2f8c452865f2ab23f0 2023-04-15 02:53:40,257 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-15 02:53:40,258 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-15 02:53:40,258 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-15 02:53:40,258 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-15 02:53:40,258 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-15 02:53:40,258 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-15 02:53:40,258 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-15 02:53:40,258 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:53:40,258 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:53:40,259 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-15 02:53:40,259 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:40,259 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-15 02:53:40,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-15 02:53:40,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:53:40,260 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:40,260 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-15 02:53:40,260 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:53:40,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-15 02:53:40,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:53:40,260 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-15 02:53:40,261 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-15 02:53:40,261 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-15 02:53:40,261 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-15 02:53:40,261 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-15 02:53:40,261 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-15 02:53:40,261 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-15 02:53:40,261 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-15 02:53:40,261 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681527213260.c44270c3412c22e2c805f466aae6ff56. 2023-04-15 02:53:40,261 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681527213260.c44270c3412c22e2c805f466aae6ff56. started... 2023-04-15 02:53:40,261 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681527213260.c44270c3412c22e2c805f466aae6ff56. as already flushing 2023-04-15 02:53:40,261 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681527213260.c44270c3412c22e2c805f466aae6ff56. 2023-04-15 02:53:40,261 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-15 02:53:40,261 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-15 02:53:40,261 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:40,261 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-15 02:53:40,262 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34681,1681527209133' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-15 02:53:40,263 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:40,263 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:53:40,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:40,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:53:40,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:53:40,263 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:53:40,263 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-15 02:53:40,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:53:40,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:53:40,264 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/c44270c3412c22e2c805f466aae6ff56/.tmp/A/cc98a063c4a940448766c211027bf809 as hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/c44270c3412c22e2c805f466aae6ff56/A/cc98a063c4a940448766c211027bf809 2023-04-15 02:53:40,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:53:40,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:40,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:53:40,266 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34681,1681527209133' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-15 02:53:40,266 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@749288fd[Count = 0] remaining members to acquire global barrier 2023-04-15 02:53:40,266 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-15 02:53:40,266 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:53:40,267 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:53:40,267 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:53:40,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:53:40,267 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-15 02:53:40,267 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-15 02:53:40,267 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34681,1681527209133' in zk 2023-04-15 02:53:40,267 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:40,268 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-15 02:53:40,271 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:40,271 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-15 02:53:40,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:40,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:53:40,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:53:40,271 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-15 02:53:40,271 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-15 02:53:40,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:53:40,272 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/c44270c3412c22e2c805f466aae6ff56/A/cc98a063c4a940448766c211027bf809, entries=200, sequenceid=163, filesize=14.1 K 2023-04-15 02:53:40,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:53:40,276 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/c44270c3412c22e2c805f466aae6ff56/.tmp/B/5e3a6ea18fae44eaa9eeb08894d78035 as hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/c44270c3412c22e2c805f466aae6ff56/B/5e3a6ea18fae44eaa9eeb08894d78035 2023-04-15 02:53:40,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:53:40,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:40,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:53:40,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:53:40,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:40,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34681,1681527209133': 2023-04-15 02:53:40,280 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34681,1681527209133' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-15 02:53:40,280 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-15 02:53:40,280 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-15 02:53:40,280 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-15 02:53:40,280 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-15 02:53:40,280 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-15 02:53:40,284 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/c44270c3412c22e2c805f466aae6ff56/B/5e3a6ea18fae44eaa9eeb08894d78035, entries=150, sequenceid=163, filesize=11.8 K 2023-04-15 02:53:40,286 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/c44270c3412c22e2c805f466aae6ff56/.tmp/C/4f01c5f98a3d4c2f8c452865f2ab23f0 as hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/c44270c3412c22e2c805f466aae6ff56/C/4f01c5f98a3d4c2f8c452865f2ab23f0 2023-04-15 02:53:40,294 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:53:40,294 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-15 02:53:40,294 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:53:40,294 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:53:40,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:53:40,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:53:40,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:53:40,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:53:40,295 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-15 02:53:40,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-15 02:53:40,295 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:40,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:53:40,295 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/c44270c3412c22e2c805f466aae6ff56/C/4f01c5f98a3d4c2f8c452865f2ab23f0, entries=150, sequenceid=163, filesize=11.8 K 2023-04-15 02:53:40,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:53:40,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:53:40,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:53:40,297 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~67.09 KB/68700, heapSize ~199.92 KB/204720, currentSize=114.05 KB/116790 for c44270c3412c22e2c805f466aae6ff56 in 1326ms, sequenceid=163, compaction requested=true 2023-04-15 02:53:40,297 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for c44270c3412c22e2c805f466aae6ff56: 2023-04-15 02:53:40,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:53:40,297 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:40,297 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-15 02:53:40,297 DEBUG [RS:0;jenkins-hbase20:34681-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-15 02:53:40,297 DEBUG [RS:0;jenkins-hbase20:34681-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-15 02:53:40,297 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-15 02:53:40,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:40,298 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-15 02:53:40,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:53:40,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:53:40,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:40,299 DEBUG [RS:0;jenkins-hbase20:34681-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 40944 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-15 02:53:40,299 DEBUG [RS:0;jenkins-hbase20:34681-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36164 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-15 02:53:40,299 DEBUG [RS:0;jenkins-hbase20:34681-shortCompactions-0] regionserver.HStore(1912): c44270c3412c22e2c805f466aae6ff56/B is initiating minor compaction (all files) 2023-04-15 02:53:40,299 DEBUG [RS:0;jenkins-hbase20:34681-longCompactions-0] regionserver.HStore(1912): c44270c3412c22e2c805f466aae6ff56/A is initiating minor compaction (all files) 2023-04-15 02:53:40,299 INFO [RS:0;jenkins-hbase20:34681-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of c44270c3412c22e2c805f466aae6ff56/B in TestAcidGuarantees,,1681527213260.c44270c3412c22e2c805f466aae6ff56. 2023-04-15 02:53:40,299 INFO [RS:0;jenkins-hbase20:34681-longCompactions-0] regionserver.HRegion(2259): Starting compaction of c44270c3412c22e2c805f466aae6ff56/A in TestAcidGuarantees,,1681527213260.c44270c3412c22e2c805f466aae6ff56. 2023-04-15 02:53:40,299 INFO [RS:0;jenkins-hbase20:34681-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/c44270c3412c22e2c805f466aae6ff56/B/2b7f5f8d4c2641089789c42009ef4cef, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/c44270c3412c22e2c805f466aae6ff56/B/4189f9dcfd404ba88dba3d8442bbb7cb, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/c44270c3412c22e2c805f466aae6ff56/B/5e3a6ea18fae44eaa9eeb08894d78035] into tmpdir=hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/c44270c3412c22e2c805f466aae6ff56/.tmp, totalSize=35.3 K 2023-04-15 02:53:40,299 INFO [RS:0;jenkins-hbase20:34681-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/c44270c3412c22e2c805f466aae6ff56/A/c6471c87fd14400580d23d2501310834, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/c44270c3412c22e2c805f466aae6ff56/A/94a6ea33c5964300bba31cc9d9175e53, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/c44270c3412c22e2c805f466aae6ff56/A/cc98a063c4a940448766c211027bf809] into tmpdir=hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/c44270c3412c22e2c805f466aae6ff56/.tmp, totalSize=40.0 K 2023-04-15 02:53:40,300 DEBUG [RS:0;jenkins-hbase20:34681-shortCompactions-0] compactions.Compactor(207): Compacting 2b7f5f8d4c2641089789c42009ef4cef, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=127, earliestPutTs=1681527217339 2023-04-15 02:53:40,300 DEBUG [RS:0;jenkins-hbase20:34681-longCompactions-0] compactions.Compactor(207): Compacting c6471c87fd14400580d23d2501310834, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=127, earliestPutTs=1681527217339 2023-04-15 02:53:40,302 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-15 02:53:40,302 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-15 02:53:40,302 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-15 02:53:40,302 DEBUG [RS:0;jenkins-hbase20:34681-longCompactions-0] compactions.Compactor(207): Compacting 94a6ea33c5964300bba31cc9d9175e53, keycount=200, bloomtype=ROW, size=14.1 K, encoding=NONE, compression=NONE, seqNum=148, earliestPutTs=1681527217684 2023-04-15 02:53:40,302 DEBUG [RS:0;jenkins-hbase20:34681-shortCompactions-0] compactions.Compactor(207): Compacting 4189f9dcfd404ba88dba3d8442bbb7cb, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=148, earliestPutTs=1681527217684 2023-04-15 02:53:40,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-15 02:53:40,303 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-15 02:53:40,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-15 02:53:40,303 DEBUG [RS:0;jenkins-hbase20:34681-shortCompactions-0] compactions.Compactor(207): Compacting 5e3a6ea18fae44eaa9eeb08894d78035, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=163, earliestPutTs=1681527218332 2023-04-15 02:53:40,304 DEBUG [RS:0;jenkins-hbase20:34681-longCompactions-0] compactions.Compactor(207): Compacting cc98a063c4a940448766c211027bf809, keycount=200, bloomtype=ROW, size=14.1 K, encoding=NONE, compression=NONE, seqNum=163, earliestPutTs=1681527218332 2023-04-15 02:53:40,304 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:40,304 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:53:40,304 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:53:40,304 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:40,304 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:53:40,304 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:53:40,304 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:53:40,305 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-15 02:53:40,305 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-15 02:53:40,305 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-15 02:53:40,305 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-15 02:53:40,305 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-15 02:53:40,305 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-15 02:53:40,322 INFO [RS:0;jenkins-hbase20:34681-shortCompactions-0] throttle.PressureAwareThroughputController(145): c44270c3412c22e2c805f466aae6ff56#B#compaction#27 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-15 02:53:40,336 INFO [RS:0;jenkins-hbase20:34681-longCompactions-0] throttle.PressureAwareThroughputController(145): c44270c3412c22e2c805f466aae6ff56#A#compaction#28 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-15 02:53:40,377 DEBUG [RS:0;jenkins-hbase20:34681-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/c44270c3412c22e2c805f466aae6ff56/.tmp/A/2f990a41b18641609c1dda35b612f63d as hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/c44270c3412c22e2c805f466aae6ff56/A/2f990a41b18641609c1dda35b612f63d 2023-04-15 02:53:40,386 INFO [RS:0;jenkins-hbase20:34681-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in c44270c3412c22e2c805f466aae6ff56/A of c44270c3412c22e2c805f466aae6ff56 into 2f990a41b18641609c1dda35b612f63d(size=12.1 K), total size for store is 12.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-15 02:53:40,386 DEBUG [RS:0;jenkins-hbase20:34681-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for c44270c3412c22e2c805f466aae6ff56: 2023-04-15 02:53:40,386 INFO [RS:0;jenkins-hbase20:34681-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681527213260.c44270c3412c22e2c805f466aae6ff56., storeName=c44270c3412c22e2c805f466aae6ff56/A, priority=13, startTime=1681527220297; duration=0sec 2023-04-15 02:53:40,386 DEBUG [RS:0;jenkins-hbase20:34681-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-15 02:53:40,386 DEBUG [RS:0;jenkins-hbase20:34681-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-15 02:53:40,389 DEBUG [RS:0;jenkins-hbase20:34681-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36164 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-15 02:53:40,390 DEBUG [RS:0;jenkins-hbase20:34681-longCompactions-0] regionserver.HStore(1912): c44270c3412c22e2c805f466aae6ff56/C is initiating minor compaction (all files) 2023-04-15 02:53:40,390 INFO [RS:0;jenkins-hbase20:34681-longCompactions-0] regionserver.HRegion(2259): Starting compaction of c44270c3412c22e2c805f466aae6ff56/C in TestAcidGuarantees,,1681527213260.c44270c3412c22e2c805f466aae6ff56. 2023-04-15 02:53:40,390 INFO [RS:0;jenkins-hbase20:34681-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/c44270c3412c22e2c805f466aae6ff56/C/f5e6234a44594733a559527c3e6a1f63, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/c44270c3412c22e2c805f466aae6ff56/C/1adba151c2d2465e8f2ae5359aad1ba5, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/c44270c3412c22e2c805f466aae6ff56/C/4f01c5f98a3d4c2f8c452865f2ab23f0] into tmpdir=hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/c44270c3412c22e2c805f466aae6ff56/.tmp, totalSize=35.3 K 2023-04-15 02:53:40,391 DEBUG [RS:0;jenkins-hbase20:34681-longCompactions-0] compactions.Compactor(207): Compacting f5e6234a44594733a559527c3e6a1f63, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=127, earliestPutTs=1681527217339 2023-04-15 02:53:40,391 DEBUG [RS:0;jenkins-hbase20:34681-longCompactions-0] compactions.Compactor(207): Compacting 1adba151c2d2465e8f2ae5359aad1ba5, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=148, earliestPutTs=1681527217684 2023-04-15 02:53:40,393 DEBUG [RS:0;jenkins-hbase20:34681-longCompactions-0] compactions.Compactor(207): Compacting 4f01c5f98a3d4c2f8c452865f2ab23f0, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=163, earliestPutTs=1681527218332 2023-04-15 02:53:40,403 INFO [RS:0;jenkins-hbase20:34681-longCompactions-0] throttle.PressureAwareThroughputController(145): c44270c3412c22e2c805f466aae6ff56#C#compaction#29 average throughput is unlimited, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-15 02:53:40,406 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-15 02:53:40,409 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-15 02:53:40,413 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-15 02:53:40,414 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-15 02:53:40,414 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-15 02:53:40,414 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-15 02:53:40,415 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-15 02:53:40,415 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-15 02:53:40,415 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-15 02:53:40,416 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:53:40,416 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:53:40,416 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-15 02:53:40,416 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:40,417 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-15 02:53:40,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-15 02:53:40,417 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:40,417 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-15 02:53:40,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:53:40,419 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:53:40,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-15 02:53:40,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:53:40,419 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-15 02:53:40,420 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-15 02:53:40,420 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-15 02:53:40,421 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-15 02:53:40,421 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-15 02:53:40,421 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-15 02:53:40,421 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-15 02:53:40,421 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-15 02:53:40,421 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681527213260.c44270c3412c22e2c805f466aae6ff56. 2023-04-15 02:53:40,421 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681527213260.c44270c3412c22e2c805f466aae6ff56. started... 2023-04-15 02:53:40,421 INFO [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing c44270c3412c22e2c805f466aae6ff56 3/3 column families, dataSize=114.05 KB heapSize=339.42 KB 2023-04-15 02:53:40,421 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c44270c3412c22e2c805f466aae6ff56, store=A 2023-04-15 02:53:40,422 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-15 02:53:40,422 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c44270c3412c22e2c805f466aae6ff56, store=B 2023-04-15 02:53:40,422 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-15 02:53:40,422 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c44270c3412c22e2c805f466aae6ff56, store=C 2023-04-15 02:53:40,422 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-15 02:53:40,427 DEBUG [RS:0;jenkins-hbase20:34681-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/c44270c3412c22e2c805f466aae6ff56/.tmp/C/c3ee48d87cbb43b6b0cc49b0a53b533f as hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/c44270c3412c22e2c805f466aae6ff56/C/c3ee48d87cbb43b6b0cc49b0a53b533f 2023-04-15 02:53:40,447 INFO [RS:0;jenkins-hbase20:34681-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in c44270c3412c22e2c805f466aae6ff56/C of c44270c3412c22e2c805f466aae6ff56 into c3ee48d87cbb43b6b0cc49b0a53b533f(size=12.1 K), total size for store is 12.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-15 02:53:40,447 DEBUG [RS:0;jenkins-hbase20:34681-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for c44270c3412c22e2c805f466aae6ff56: 2023-04-15 02:53:40,447 INFO [RS:0;jenkins-hbase20:34681-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681527213260.c44270c3412c22e2c805f466aae6ff56., storeName=c44270c3412c22e2c805f466aae6ff56/C, priority=13, startTime=1681527220298; duration=0sec 2023-04-15 02:53:40,447 DEBUG [RS:0;jenkins-hbase20:34681-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-15 02:53:40,777 DEBUG [RS:0;jenkins-hbase20:34681-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/c44270c3412c22e2c805f466aae6ff56/.tmp/B/00e29523c81a4a4b8618ef27d74d4385 as hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/c44270c3412c22e2c805f466aae6ff56/B/00e29523c81a4a4b8618ef27d74d4385 2023-04-15 02:53:40,788 INFO [RS:0;jenkins-hbase20:34681-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in c44270c3412c22e2c805f466aae6ff56/B of c44270c3412c22e2c805f466aae6ff56 into 00e29523c81a4a4b8618ef27d74d4385(size=12.1 K), total size for store is 12.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-15 02:53:40,788 DEBUG [RS:0;jenkins-hbase20:34681-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for c44270c3412c22e2c805f466aae6ff56: 2023-04-15 02:53:40,788 INFO [RS:0;jenkins-hbase20:34681-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681527213260.c44270c3412c22e2c805f466aae6ff56., storeName=c44270c3412c22e2c805f466aae6ff56/B, priority=13, startTime=1681527220297; duration=0sec 2023-04-15 02:53:40,788 DEBUG [RS:0;jenkins-hbase20:34681-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-15 02:53:40,856 INFO [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=184 (bloomFilter=true), to=hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/c44270c3412c22e2c805f466aae6ff56/.tmp/A/a4e8adecaa744cf8bdc2810ae3aac070 2023-04-15 02:53:40,874 INFO [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=184 (bloomFilter=true), to=hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/c44270c3412c22e2c805f466aae6ff56/.tmp/B/71f84bdaa00940eda909883ab8648894 2023-04-15 02:53:40,893 INFO [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=184 (bloomFilter=true), to=hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/c44270c3412c22e2c805f466aae6ff56/.tmp/C/845decfbcaff4e56a8eb1e0a5db76d11 2023-04-15 02:53:40,900 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/c44270c3412c22e2c805f466aae6ff56/.tmp/A/a4e8adecaa744cf8bdc2810ae3aac070 as hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/c44270c3412c22e2c805f466aae6ff56/A/a4e8adecaa744cf8bdc2810ae3aac070 2023-04-15 02:53:40,907 INFO [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/c44270c3412c22e2c805f466aae6ff56/A/a4e8adecaa744cf8bdc2810ae3aac070, entries=150, sequenceid=184, filesize=11.8 K 2023-04-15 02:53:40,909 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/c44270c3412c22e2c805f466aae6ff56/.tmp/B/71f84bdaa00940eda909883ab8648894 as hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/c44270c3412c22e2c805f466aae6ff56/B/71f84bdaa00940eda909883ab8648894 2023-04-15 02:53:40,924 INFO [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/c44270c3412c22e2c805f466aae6ff56/B/71f84bdaa00940eda909883ab8648894, entries=150, sequenceid=184, filesize=11.8 K 2023-04-15 02:53:40,928 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/c44270c3412c22e2c805f466aae6ff56/.tmp/C/845decfbcaff4e56a8eb1e0a5db76d11 as hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/c44270c3412c22e2c805f466aae6ff56/C/845decfbcaff4e56a8eb1e0a5db76d11 2023-04-15 02:53:40,938 INFO [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/c44270c3412c22e2c805f466aae6ff56/C/845decfbcaff4e56a8eb1e0a5db76d11, entries=150, sequenceid=184, filesize=11.8 K 2023-04-15 02:53:40,940 INFO [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~114.05 KB/116790, heapSize ~339.38 KB/347520, currentSize=0 B/0 for c44270c3412c22e2c805f466aae6ff56 in 519ms, sequenceid=184, compaction requested=false 2023-04-15 02:53:40,940 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for c44270c3412c22e2c805f466aae6ff56: 2023-04-15 02:53:40,940 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681527213260.c44270c3412c22e2c805f466aae6ff56. 2023-04-15 02:53:40,940 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-15 02:53:40,943 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-15 02:53:40,943 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:40,943 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-15 02:53:40,943 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34681,1681527209133' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-15 02:53:40,944 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:40,944 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:53:40,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:40,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:53:40,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:53:40,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:53:40,945 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:53:40,945 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-15 02:53:40,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:53:40,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:53:40,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:40,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:53:40,947 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34681,1681527209133' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-15 02:53:40,947 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-15 02:53:40,947 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:53:40,948 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@55b4b1[Count = 0] remaining members to acquire global barrier 2023-04-15 02:53:40,948 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:53:40,949 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:53:40,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:53:40,949 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-15 02:53:40,949 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-15 02:53:40,949 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34681,1681527209133' in zk 2023-04-15 02:53:40,949 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:40,949 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-15 02:53:40,950 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:40,950 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-15 02:53:40,950 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-15 02:53:40,950 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-15 02:53:40,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:40,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:53:40,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:53:40,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:53:40,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:53:40,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:53:40,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:40,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:53:40,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:53:40,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:40,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34681,1681527209133': 2023-04-15 02:53:40,953 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34681,1681527209133' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-15 02:53:40,953 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-15 02:53:40,953 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-15 02:53:40,953 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-15 02:53:40,953 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-15 02:53:40,953 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-15 02:53:40,954 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:53:40,954 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:53:40,954 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-15 02:53:40,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:53:40,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:53:40,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:53:40,954 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:53:40,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:53:40,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:53:40,955 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:40,955 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-15 02:53:40,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-15 02:53:40,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:53:40,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:53:40,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:53:40,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:53:40,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:40,957 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:40,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:53:40,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:53:40,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:40,959 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-15 02:53:40,960 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-15 02:53:40,960 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-15 02:53:40,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-15 02:53:40,960 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-15 02:53:40,959 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:40,960 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:53:40,960 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-15 02:53:40,960 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:53:40,960 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:40,960 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:53:40,960 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-15 02:53:40,960 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-15 02:53:40,960 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:53:40,960 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:53:40,960 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-15 02:53:40,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-15 02:53:40,960 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-15 02:53:40,960 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-15 02:53:41,061 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-15 02:53:41,061 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-15 02:53:41,063 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-15 02:53:41,064 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-15 02:53:41,064 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-15 02:53:41,064 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-15 02:53:41,064 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-15 02:53:41,064 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-15 02:53:41,064 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-15 02:53:41,065 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:53:41,065 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:53:41,066 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-15 02:53:41,066 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:41,066 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-15 02:53:41,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-15 02:53:41,066 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:41,066 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-15 02:53:41,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:53:41,066 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:53:41,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-15 02:53:41,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:53:41,067 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-15 02:53:41,067 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-15 02:53:41,067 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-15 02:53:41,067 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-15 02:53:41,067 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-15 02:53:41,067 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-15 02:53:41,067 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-15 02:53:41,067 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-15 02:53:41,067 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681527213260.c44270c3412c22e2c805f466aae6ff56. 2023-04-15 02:53:41,068 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681527213260.c44270c3412c22e2c805f466aae6ff56. started... 2023-04-15 02:53:41,068 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for c44270c3412c22e2c805f466aae6ff56: 2023-04-15 02:53:41,068 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681527213260.c44270c3412c22e2c805f466aae6ff56. 2023-04-15 02:53:41,068 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-15 02:53:41,068 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-15 02:53:41,068 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:41,068 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-15 02:53:41,068 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34681,1681527209133' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-15 02:53:41,069 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:41,069 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:53:41,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:41,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:53:41,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:53:41,069 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:53:41,069 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-15 02:53:41,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:53:41,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:53:41,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:53:41,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:41,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:53:41,071 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34681,1681527209133' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-15 02:53:41,071 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5e8df762[Count = 0] remaining members to acquire global barrier 2023-04-15 02:53:41,071 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-15 02:53:41,071 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:53:41,071 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:53:41,071 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:53:41,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:53:41,072 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:41,072 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-15 02:53:41,072 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-15 02:53:41,072 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-15 02:53:41,072 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34681,1681527209133' in zk 2023-04-15 02:53:41,075 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-15 02:53:41,075 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:41,075 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-15 02:53:41,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:41,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:53:41,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:53:41,075 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-15 02:53:41,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:53:41,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:53:41,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:53:41,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:41,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:53:41,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:53:41,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:41,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34681,1681527209133': 2023-04-15 02:53:41,077 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34681,1681527209133' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-15 02:53:41,077 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-15 02:53:41,077 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-15 02:53:41,077 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-15 02:53:41,077 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-15 02:53:41,077 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-15 02:53:41,078 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:53:41,078 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:53:41,078 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:53:41,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:53:41,078 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-15 02:53:41,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:53:41,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:53:41,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:53:41,079 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-15 02:53:41,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-15 02:53:41,079 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:41,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:53:41,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:53:41,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:53:41,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:53:41,080 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:41,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:53:41,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:41,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:53:41,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:53:41,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:41,081 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:41,081 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:53:41,082 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:53:41,082 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-15 02:53:41,082 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-15 02:53:41,082 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-15 02:53:41,082 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-15 02:53:41,082 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-15 02:53:41,082 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-15 02:53:41,082 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:41,082 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:53:41,082 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:53:41,082 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:53:41,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-15 02:53:41,082 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-15 02:53:41,082 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-15 02:53:41,082 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-15 02:53:41,082 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-15 02:53:41,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-15 02:53:41,182 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-15 02:53:41,183 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-15 02:53:41,184 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34681] regionserver.HRegion(9158): Flush requested on c44270c3412c22e2c805f466aae6ff56 2023-04-15 02:53:41,184 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing c44270c3412c22e2c805f466aae6ff56 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-15 02:53:41,185 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c44270c3412c22e2c805f466aae6ff56, store=A 2023-04-15 02:53:41,185 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-15 02:53:41,185 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c44270c3412c22e2c805f466aae6ff56, store=B 2023-04-15 02:53:41,185 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-15 02:53:41,185 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c44270c3412c22e2c805f466aae6ff56, store=C 2023-04-15 02:53:41,185 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-15 02:53:41,185 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-15 02:53:41,187 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-15 02:53:41,187 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-15 02:53:41,187 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-15 02:53:41,187 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-15 02:53:41,187 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-15 02:53:41,187 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-15 02:53:41,188 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:53:41,188 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:53:41,189 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:41,189 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-15 02:53:41,190 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-15 02:53:41,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-15 02:53:41,190 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:41,190 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-15 02:53:41,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:53:41,191 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:53:41,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-15 02:53:41,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:53:41,192 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-15 02:53:41,198 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-15 02:53:41,199 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-15 02:53:41,199 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-15 02:53:41,199 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-15 02:53:41,199 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-15 02:53:41,199 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-15 02:53:41,199 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-15 02:53:41,199 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681527213260.c44270c3412c22e2c805f466aae6ff56. 2023-04-15 02:53:41,199 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681527213260.c44270c3412c22e2c805f466aae6ff56. started... 2023-04-15 02:53:41,199 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681527213260.c44270c3412c22e2c805f466aae6ff56. as already flushing 2023-04-15 02:53:41,199 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681527213260.c44270c3412c22e2c805f466aae6ff56. 2023-04-15 02:53:41,199 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-15 02:53:41,199 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-15 02:53:41,199 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:41,199 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-15 02:53:41,199 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34681,1681527209133' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-15 02:53:41,202 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:41,202 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:53:41,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:41,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:53:41,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:53:41,203 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:53:41,203 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-15 02:53:41,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:53:41,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:53:41,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:53:41,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:41,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:53:41,205 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34681,1681527209133' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-15 02:53:41,205 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4c26bceb[Count = 0] remaining members to acquire global barrier 2023-04-15 02:53:41,205 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-15 02:53:41,205 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:53:41,206 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:53:41,206 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:53:41,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:53:41,206 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-15 02:53:41,206 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-15 02:53:41,206 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34681,1681527209133' in zk 2023-04-15 02:53:41,206 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:41,206 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-15 02:53:41,208 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c44270c3412c22e2c805f466aae6ff56, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:53:41,208 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34681] ipc.CallRunner(144): callId: 77 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:38530 deadline: 1681527281204, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c44270c3412c22e2c805f466aae6ff56, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:41,208 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1681527221197, max timestamp=1681527221206Num uniques 859; 2023-04-15 02:53:41,208 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-4] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-04-15 02:53:41,209 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1681527221197, max timestamp=1681527221206Num uniques 859; 2023-04-15 02:53:41,209 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1681527221197, max timestamp=1681527221206Num uniques 859; 2023-04-15 02:53:41,209 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-3] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-04-15 02:53:41,209 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-04-15 02:53:41,209 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-15 02:53:41,209 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:41,210 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-15 02:53:41,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:41,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:53:41,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:53:41,210 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-15 02:53:41,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:53:41,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:53:41,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:53:41,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:41,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:53:41,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:53:41,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:41,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34681,1681527209133': 2023-04-15 02:53:41,213 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34681,1681527209133' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-15 02:53:41,213 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-15 02:53:41,213 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-15 02:53:41,213 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-15 02:53:41,213 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-15 02:53:41,213 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-15 02:53:41,215 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:53:41,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:53:41,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:53:41,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:53:41,215 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:53:41,215 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-15 02:53:41,215 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:53:41,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:53:41,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:53:41,215 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-15 02:53:41,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:53:41,215 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:41,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-15 02:53:41,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:53:41,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:53:41,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:53:41,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:41,217 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:41,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:53:41,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:53:41,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:41,219 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-15 02:53:41,219 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-15 02:53:41,219 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:41,220 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:53:41,220 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:53:41,220 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:41,220 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-15 02:53:41,220 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-15 02:53:41,219 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-15 02:53:41,220 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:53:41,220 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-15 02:53:41,220 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:53:41,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-15 02:53:41,220 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-15 02:53:41,220 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-15 02:53:41,220 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:53:41,220 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-15 02:53:41,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-15 02:53:41,229 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c44270c3412c22e2c805f466aae6ff56, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:53:41,230 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34681] ipc.CallRunner(144): callId: 83 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:38504 deadline: 1681527281226, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c44270c3412c22e2c805f466aae6ff56, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:41,230 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c44270c3412c22e2c805f466aae6ff56, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:53:41,230 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34681] ipc.CallRunner(144): callId: 81 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:38542 deadline: 1681527281228, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c44270c3412c22e2c805f466aae6ff56, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:41,232 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c44270c3412c22e2c805f466aae6ff56, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:53:41,232 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34681] ipc.CallRunner(144): callId: 84 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:38520 deadline: 1681527281230, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c44270c3412c22e2c805f466aae6ff56, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:41,310 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c44270c3412c22e2c805f466aae6ff56, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:53:41,310 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34681] ipc.CallRunner(144): callId: 79 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:38530 deadline: 1681527281309, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c44270c3412c22e2c805f466aae6ff56, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:41,320 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-15 02:53:41,321 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-15 02:53:41,322 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-15 02:53:41,323 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-15 02:53:41,323 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-15 02:53:41,323 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-15 02:53:41,323 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-15 02:53:41,323 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-15 02:53:41,324 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-15 02:53:41,324 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:53:41,325 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:53:41,326 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:41,326 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-15 02:53:41,326 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-15 02:53:41,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-15 02:53:41,326 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:41,326 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-15 02:53:41,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:53:41,326 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:53:41,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-15 02:53:41,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:53:41,327 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-15 02:53:41,327 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-15 02:53:41,327 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-15 02:53:41,327 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-15 02:53:41,327 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-15 02:53:41,327 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-15 02:53:41,327 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-15 02:53:41,327 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-15 02:53:41,327 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681527213260.c44270c3412c22e2c805f466aae6ff56. 2023-04-15 02:53:41,327 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681527213260.c44270c3412c22e2c805f466aae6ff56. started... 2023-04-15 02:53:41,327 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681527213260.c44270c3412c22e2c805f466aae6ff56. as already flushing 2023-04-15 02:53:41,327 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681527213260.c44270c3412c22e2c805f466aae6ff56. 2023-04-15 02:53:41,327 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-15 02:53:41,327 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-15 02:53:41,328 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:41,328 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-15 02:53:41,328 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34681,1681527209133' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-15 02:53:41,328 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:41,328 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:53:41,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:41,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:53:41,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:53:41,329 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:53:41,329 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-15 02:53:41,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:53:41,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:53:41,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:53:41,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:41,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:53:41,330 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34681,1681527209133' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-15 02:53:41,330 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2c9300f2[Count = 0] remaining members to acquire global barrier 2023-04-15 02:53:41,330 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-15 02:53:41,330 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:53:41,331 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:53:41,331 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:53:41,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:53:41,331 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-15 02:53:41,331 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-15 02:53:41,331 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34681,1681527209133' in zk 2023-04-15 02:53:41,332 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-15 02:53:41,332 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-15 02:53:41,332 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-15 02:53:41,333 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:41,333 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34681,1681527209133' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-15 02:53:41,333 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-15 02:53:41,333 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-15 02:53:41,333 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-15 02:53:41,333 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-15 02:53:41,333 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-15 02:53:41,333 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-15 02:53:41,334 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:53:41,334 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:53:41,334 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-15 02:53:41,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:53:41,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:53:41,334 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:53:41,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:53:41,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:53:41,335 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:41,335 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-15 02:53:41,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:53:41,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-15 02:53:41,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:53:41,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:53:41,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:53:41,336 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:41,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:53:41,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:41,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:53:41,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:53:41,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:41,338 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:41,338 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:53:41,338 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:53:41,338 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:41,338 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-15 02:53:41,338 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-15 02:53:41,338 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-15 02:53:41,338 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-15 02:53:41,338 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-15 02:53:41,338 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:53:41,338 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:53:41,338 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:53:41,338 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-15 02:53:41,338 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-15 02:53:41,338 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-15 02:53:41,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-15 02:53:41,338 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-15 02:53:41,338 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c44270c3412c22e2c805f466aae6ff56, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:53:41,339 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-15 02:53:41,339 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34681] ipc.CallRunner(144): callId: 85 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:38504 deadline: 1681527281334, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c44270c3412c22e2c805f466aae6ff56, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:41,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-15 02:53:41,339 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c44270c3412c22e2c805f466aae6ff56, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:53:41,339 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34681] ipc.CallRunner(144): callId: 86 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:38520 deadline: 1681527281334, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c44270c3412c22e2c805f466aae6ff56, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:41,339 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c44270c3412c22e2c805f466aae6ff56, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:53:41,339 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34681] ipc.CallRunner(144): callId: 83 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:38542 deadline: 1681527281334, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c44270c3412c22e2c805f466aae6ff56, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:41,438 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-15 02:53:41,439 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-15 02:53:41,442 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-15 02:53:41,444 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-15 02:53:41,444 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-15 02:53:41,444 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-15 02:53:41,444 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-15 02:53:41,444 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-15 02:53:41,444 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-15 02:53:41,444 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:53:41,444 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:53:41,445 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:41,445 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-15 02:53:41,445 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-15 02:53:41,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-15 02:53:41,445 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:41,446 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-15 02:53:41,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:53:41,446 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:53:41,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-15 02:53:41,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:53:41,446 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-15 02:53:41,447 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-15 02:53:41,447 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-15 02:53:41,447 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-15 02:53:41,447 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-15 02:53:41,447 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-15 02:53:41,447 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-15 02:53:41,447 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-15 02:53:41,447 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681527213260.c44270c3412c22e2c805f466aae6ff56. 2023-04-15 02:53:41,447 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681527213260.c44270c3412c22e2c805f466aae6ff56. started... 2023-04-15 02:53:41,447 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681527213260.c44270c3412c22e2c805f466aae6ff56. as already flushing 2023-04-15 02:53:41,447 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681527213260.c44270c3412c22e2c805f466aae6ff56. 2023-04-15 02:53:41,447 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-15 02:53:41,447 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-15 02:53:41,447 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:41,447 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-15 02:53:41,447 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34681,1681527209133' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-15 02:53:41,448 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:41,448 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:53:41,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:41,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:53:41,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:53:41,449 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:53:41,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:53:41,449 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-15 02:53:41,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:53:41,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:53:41,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:41,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:53:41,450 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34681,1681527209133' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-15 02:53:41,450 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@14ec9ed[Count = 0] remaining members to acquire global barrier 2023-04-15 02:53:41,450 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-15 02:53:41,450 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:53:41,451 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:53:41,451 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:53:41,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:53:41,451 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-15 02:53:41,451 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-15 02:53:41,451 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34681,1681527209133' in zk 2023-04-15 02:53:41,451 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:41,451 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-15 02:53:41,451 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-15 02:53:41,451 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-15 02:53:41,452 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:41,452 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-15 02:53:41,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:41,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:53:41,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:53:41,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:53:41,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:53:41,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:53:41,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:41,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:53:41,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:53:41,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:41,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34681,1681527209133': 2023-04-15 02:53:41,456 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34681,1681527209133' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-15 02:53:41,456 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-15 02:53:41,456 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-15 02:53:41,456 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-15 02:53:41,456 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-15 02:53:41,456 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-15 02:53:41,460 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:53:41,461 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-15 02:53:41,461 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:53:41,461 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:53:41,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:53:41,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:53:41,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:53:41,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:53:41,461 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-15 02:53:41,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-15 02:53:41,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:53:41,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:53:41,462 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:41,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:53:41,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:53:41,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:53:41,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:41,463 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:41,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:53:41,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:53:41,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:41,465 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:41,465 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-15 02:53:41,465 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:53:41,465 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-15 02:53:41,465 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:53:41,465 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-15 02:53:41,465 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-15 02:53:41,465 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-15 02:53:41,465 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:41,465 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-15 02:53:41,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-15 02:53:41,465 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:53:41,465 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-15 02:53:41,465 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:53:41,465 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-15 02:53:41,466 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:53:41,466 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-15 02:53:41,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-15 02:53:41,513 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c44270c3412c22e2c805f466aae6ff56, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:53:41,513 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34681] ipc.CallRunner(144): callId: 81 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:38530 deadline: 1681527281512, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c44270c3412c22e2c805f466aae6ff56, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:41,542 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c44270c3412c22e2c805f466aae6ff56, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:53:41,543 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34681] ipc.CallRunner(144): callId: 85 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:38542 deadline: 1681527281541, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c44270c3412c22e2c805f466aae6ff56, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:41,543 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c44270c3412c22e2c805f466aae6ff56, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:53:41,543 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34681] ipc.CallRunner(144): callId: 87 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:38504 deadline: 1681527281541, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c44270c3412c22e2c805f466aae6ff56, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:41,543 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c44270c3412c22e2c805f466aae6ff56, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:53:41,543 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34681] ipc.CallRunner(144): callId: 88 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:38520 deadline: 1681527281542, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c44270c3412c22e2c805f466aae6ff56, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:41,554 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c44270c3412c22e2c805f466aae6ff56, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:53:41,555 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34681] ipc.CallRunner(144): callId: 47 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:38540 deadline: 1681527281552, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c44270c3412c22e2c805f466aae6ff56, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:41,556 DEBUG [Thread-143] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4195 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c44270c3412c22e2c805f466aae6ff56, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_2' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1681527213260.c44270c3412c22e2c805f466aae6ff56., hostname=jenkins-hbase20.apache.org,34681,1681527209133, seqNum=2, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c44270c3412c22e2c805f466aae6ff56, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor31.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c44270c3412c22e2c805f466aae6ff56, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-15 02:53:41,566 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-15 02:53:41,566 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-15 02:53:41,570 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-15 02:53:41,571 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-15 02:53:41,572 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-15 02:53:41,572 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-15 02:53:41,572 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-15 02:53:41,572 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-15 02:53:41,572 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-15 02:53:41,572 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:53:41,572 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:53:41,573 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-15 02:53:41,573 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:41,573 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-15 02:53:41,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-15 02:53:41,573 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:41,573 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-15 02:53:41,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:53:41,574 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:53:41,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-15 02:53:41,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:53:41,574 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-15 02:53:41,575 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-15 02:53:41,575 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-15 02:53:41,575 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-15 02:53:41,575 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-15 02:53:41,575 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-15 02:53:41,575 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-15 02:53:41,575 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-15 02:53:41,575 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681527213260.c44270c3412c22e2c805f466aae6ff56. 2023-04-15 02:53:41,575 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681527213260.c44270c3412c22e2c805f466aae6ff56. started... 2023-04-15 02:53:41,575 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681527213260.c44270c3412c22e2c805f466aae6ff56. as already flushing 2023-04-15 02:53:41,575 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681527213260.c44270c3412c22e2c805f466aae6ff56. 2023-04-15 02:53:41,575 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-15 02:53:41,575 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-15 02:53:41,575 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:41,575 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-15 02:53:41,575 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34681,1681527209133' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-15 02:53:41,578 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:41,578 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:53:41,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:41,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:53:41,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:53:41,578 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:53:41,578 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-15 02:53:41,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:53:41,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:53:41,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:53:41,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:41,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:53:41,580 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34681,1681527209133' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-15 02:53:41,580 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-15 02:53:41,580 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@30c91841[Count = 0] remaining members to acquire global barrier 2023-04-15 02:53:41,580 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:53:41,580 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:53:41,581 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:53:41,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:53:41,581 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-15 02:53:41,581 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:41,581 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-15 02:53:41,581 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-15 02:53:41,581 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34681,1681527209133' in zk 2023-04-15 02:53:41,582 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:41,582 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-15 02:53:41,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:41,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:53:41,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:53:41,582 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-15 02:53:41,582 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-15 02:53:41,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:53:41,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:53:41,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:53:41,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:41,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:53:41,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:53:41,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:41,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34681,1681527209133': 2023-04-15 02:53:41,586 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34681,1681527209133' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-15 02:53:41,586 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-15 02:53:41,586 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-15 02:53:41,586 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-15 02:53:41,586 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-15 02:53:41,586 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-15 02:53:41,594 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:53:41,594 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:53:41,594 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:53:41,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:53:41,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:53:41,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:53:41,594 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-15 02:53:41,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:53:41,594 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:41,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:53:41,594 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-15 02:53:41,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-15 02:53:41,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:53:41,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:53:41,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:53:41,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:53:41,595 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:41,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:41,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:53:41,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:53:41,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:41,598 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:41,598 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:53:41,598 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:53:41,598 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:41,598 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:53:41,598 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:53:41,598 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:53:41,598 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-15 02:53:41,598 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-15 02:53:41,598 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-15 02:53:41,598 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-15 02:53:41,599 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-15 02:53:41,599 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-15 02:53:41,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-15 02:53:41,599 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-15 02:53:41,599 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-15 02:53:41,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-15 02:53:41,599 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-15 02:53:41,599 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-15 02:53:41,610 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=197 (bloomFilter=true), to=hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/c44270c3412c22e2c805f466aae6ff56/.tmp/A/738be694a3d34996a54af27f4b8b76f0 2023-04-15 02:53:41,699 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-15 02:53:41,700 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-15 02:53:41,702 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-15 02:53:41,703 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-15 02:53:41,703 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-15 02:53:41,703 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-15 02:53:41,703 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-15 02:53:41,703 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-15 02:53:41,703 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-15 02:53:41,703 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:53:41,703 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:53:41,704 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-15 02:53:41,704 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:41,704 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-15 02:53:41,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-15 02:53:41,705 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:41,705 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-15 02:53:41,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:53:41,705 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:53:41,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-15 02:53:41,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:53:41,705 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-15 02:53:41,706 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-15 02:53:41,706 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-15 02:53:41,706 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-15 02:53:41,706 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-15 02:53:41,706 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-15 02:53:41,710 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-15 02:53:41,710 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681527213260.c44270c3412c22e2c805f466aae6ff56. 2023-04-15 02:53:41,710 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-15 02:53:41,710 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681527213260.c44270c3412c22e2c805f466aae6ff56. started... 2023-04-15 02:53:41,710 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681527213260.c44270c3412c22e2c805f466aae6ff56. as already flushing 2023-04-15 02:53:41,710 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681527213260.c44270c3412c22e2c805f466aae6ff56. 2023-04-15 02:53:41,710 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-15 02:53:41,710 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-15 02:53:41,710 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:41,710 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-15 02:53:41,710 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34681,1681527209133' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-15 02:53:41,715 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:41,715 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:53:41,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:41,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:53:41,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:53:41,715 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:53:41,716 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-15 02:53:41,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:53:41,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:53:41,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:53:41,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:41,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:53:41,717 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34681,1681527209133' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-15 02:53:41,717 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@49712b57[Count = 0] remaining members to acquire global barrier 2023-04-15 02:53:41,717 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-15 02:53:41,717 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:53:41,719 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:53:41,719 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:53:41,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:53:41,719 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-15 02:53:41,719 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:41,719 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-15 02:53:41,719 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-15 02:53:41,719 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34681,1681527209133' in zk 2023-04-15 02:53:41,720 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:41,720 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-15 02:53:41,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:41,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:53:41,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:53:41,720 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-15 02:53:41,720 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-15 02:53:41,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:53:41,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:53:41,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:53:41,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:41,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:53:41,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:53:41,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:41,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34681,1681527209133': 2023-04-15 02:53:41,722 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34681,1681527209133' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-15 02:53:41,722 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-15 02:53:41,722 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-15 02:53:41,722 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-15 02:53:41,722 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-15 02:53:41,722 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-15 02:53:41,724 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:53:41,724 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:53:41,724 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:53:41,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:53:41,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:53:41,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:53:41,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:53:41,724 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-15 02:53:41,724 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-15 02:53:41,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-15 02:53:41,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:53:41,726 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:41,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:53:41,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:53:41,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:53:41,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:53:41,734 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:41,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:41,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:53:41,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:53:41,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:41,735 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:41,735 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-15 02:53:41,735 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-15 02:53:41,736 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-15 02:53:41,735 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:53:41,736 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-15 02:53:41,736 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-15 02:53:41,736 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:53:41,736 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-15 02:53:41,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-15 02:53:41,736 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:41,736 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:53:41,736 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:53:41,736 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:53:41,736 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-15 02:53:41,736 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-15 02:53:41,736 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-15 02:53:41,736 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-15 02:53:41,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-15 02:53:41,815 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c44270c3412c22e2c805f466aae6ff56, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:53:41,815 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34681] ipc.CallRunner(144): callId: 83 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:38530 deadline: 1681527281814, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c44270c3412c22e2c805f466aae6ff56, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:41,836 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-15 02:53:41,837 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-15 02:53:41,840 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-15 02:53:41,841 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-15 02:53:41,841 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-15 02:53:41,841 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-15 02:53:41,841 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-15 02:53:41,841 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-15 02:53:41,841 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-15 02:53:41,841 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:53:41,841 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:53:41,842 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-15 02:53:41,843 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:41,843 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-15 02:53:41,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-15 02:53:41,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:53:41,843 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:41,843 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-15 02:53:41,843 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:53:41,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-15 02:53:41,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:53:41,844 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-15 02:53:41,844 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-15 02:53:41,844 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-15 02:53:41,844 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-15 02:53:41,844 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-15 02:53:41,844 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-15 02:53:41,844 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-15 02:53:41,844 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-15 02:53:41,844 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681527213260.c44270c3412c22e2c805f466aae6ff56. 2023-04-15 02:53:41,844 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681527213260.c44270c3412c22e2c805f466aae6ff56. started... 2023-04-15 02:53:41,845 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681527213260.c44270c3412c22e2c805f466aae6ff56. as already flushing 2023-04-15 02:53:41,845 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681527213260.c44270c3412c22e2c805f466aae6ff56. 2023-04-15 02:53:41,845 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-15 02:53:41,845 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-15 02:53:41,845 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:41,845 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-15 02:53:41,845 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34681,1681527209133' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-15 02:53:41,846 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:41,846 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:53:41,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:41,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:53:41,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:53:41,846 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c44270c3412c22e2c805f466aae6ff56, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:53:41,847 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34681] ipc.CallRunner(144): callId: 87 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:38542 deadline: 1681527281846, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c44270c3412c22e2c805f466aae6ff56, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:41,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:53:41,847 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:53:41,847 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-15 02:53:41,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:53:41,847 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c44270c3412c22e2c805f466aae6ff56, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:53:41,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:53:41,847 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34681] ipc.CallRunner(144): callId: 89 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:38504 deadline: 1681527281847, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c44270c3412c22e2c805f466aae6ff56, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:41,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:41,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:53:41,848 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34681,1681527209133' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-15 02:53:41,848 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c44270c3412c22e2c805f466aae6ff56, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:53:41,848 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-15 02:53:41,848 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34681] ipc.CallRunner(144): callId: 90 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:38520 deadline: 1681527281847, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c44270c3412c22e2c805f466aae6ff56, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:41,848 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7d51361b[Count = 0] remaining members to acquire global barrier 2023-04-15 02:53:41,848 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:53:41,849 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:53:41,850 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:53:41,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:53:41,850 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:41,850 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-15 02:53:41,850 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-15 02:53:41,850 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-15 02:53:41,850 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34681,1681527209133' in zk 2023-04-15 02:53:41,851 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:41,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:41,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:53:41,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:53:41,851 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-15 02:53:41,851 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-15 02:53:41,851 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-15 02:53:41,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:53:41,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:53:41,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:53:41,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:41,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:53:41,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:53:41,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:41,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34681,1681527209133': 2023-04-15 02:53:41,853 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34681,1681527209133' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-15 02:53:41,853 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-15 02:53:41,853 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-15 02:53:41,853 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-15 02:53:41,853 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-15 02:53:41,853 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-15 02:53:41,854 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:53:41,855 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:53:41,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:53:41,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:53:41,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:53:41,855 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-15 02:53:41,855 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:53:41,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:53:41,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:53:41,855 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:41,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:53:41,857 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-15 02:53:41,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-15 02:53:41,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:53:41,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:53:41,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:53:41,857 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:41,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:41,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:53:41,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:53:41,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:41,859 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-15 02:53:41,859 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-15 02:53:41,859 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-15 02:53:41,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-15 02:53:41,859 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-15 02:53:41,859 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-15 02:53:41,859 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-15 02:53:41,859 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:41,860 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:53:41,860 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:53:41,860 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:41,860 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:53:41,860 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:53:41,860 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:53:41,859 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-15 02:53:41,860 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-15 02:53:41,860 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-15 02:53:41,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-15 02:53:41,860 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-15 02:53:41,960 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-15 02:53:41,961 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-15 02:53:41,963 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-15 02:53:41,963 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-15 02:53:41,964 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-15 02:53:41,964 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-15 02:53:41,964 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-15 02:53:41,964 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-15 02:53:41,964 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-15 02:53:41,964 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:53:41,964 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:53:41,965 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-15 02:53:41,965 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:41,965 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-15 02:53:41,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-15 02:53:41,965 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:41,965 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-15 02:53:41,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:53:41,966 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:53:41,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-15 02:53:41,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:53:41,966 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-15 02:53:41,966 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-15 02:53:41,966 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-15 02:53:41,966 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-15 02:53:41,967 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-15 02:53:41,967 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-15 02:53:41,967 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-15 02:53:41,967 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-15 02:53:41,967 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681527213260.c44270c3412c22e2c805f466aae6ff56. 2023-04-15 02:53:41,967 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681527213260.c44270c3412c22e2c805f466aae6ff56. started... 2023-04-15 02:53:41,967 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681527213260.c44270c3412c22e2c805f466aae6ff56. as already flushing 2023-04-15 02:53:41,967 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681527213260.c44270c3412c22e2c805f466aae6ff56. 2023-04-15 02:53:41,967 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-15 02:53:41,967 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-15 02:53:41,967 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:41,967 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-15 02:53:41,967 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34681,1681527209133' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-15 02:53:41,968 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:41,968 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:53:41,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:41,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:53:41,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:53:41,969 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:53:41,969 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-15 02:53:41,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:53:41,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:53:41,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:53:41,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:41,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:53:41,970 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34681,1681527209133' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-15 02:53:41,970 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@a2799fd[Count = 0] remaining members to acquire global barrier 2023-04-15 02:53:41,970 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-15 02:53:41,970 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:53:41,971 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:53:41,971 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:53:41,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:53:41,971 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-15 02:53:41,971 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-15 02:53:41,971 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34681,1681527209133' in zk 2023-04-15 02:53:41,971 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:41,971 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-15 02:53:41,972 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:41,972 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-15 02:53:41,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:41,973 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-15 02:53:41,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:53:41,973 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-15 02:53:41,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:53:41,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:53:41,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:53:41,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:53:41,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:41,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:53:41,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:53:41,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:41,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34681,1681527209133': 2023-04-15 02:53:41,975 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34681,1681527209133' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-15 02:53:41,975 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-15 02:53:41,975 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-15 02:53:41,975 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-15 02:53:41,975 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-15 02:53:41,975 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-15 02:53:41,976 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:53:41,976 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-15 02:53:41,976 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:53:41,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:53:41,976 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:53:41,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:53:41,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:53:41,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:53:41,976 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-15 02:53:41,976 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:41,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-15 02:53:41,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:53:41,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:53:41,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:53:41,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:53:41,979 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:41,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:53:41,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:41,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:53:41,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:53:41,981 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-15 02:53:41,981 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:41,981 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-15 02:53:41,981 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:53:41,981 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-15 02:53:41,981 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:53:41,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-15 02:53:41,981 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:41,981 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:53:41,981 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:53:41,982 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-15 02:53:41,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-15 02:53:41,983 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:53:41,983 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-15 02:53:41,983 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-15 02:53:41,984 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-15 02:53:41,984 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-15 02:53:41,984 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-15 02:53:41,984 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-15 02:53:42,053 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=197 (bloomFilter=true), to=hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/c44270c3412c22e2c805f466aae6ff56/.tmp/B/e91a0eec44524aaea30d1409f1d7aac8 2023-04-15 02:53:42,066 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=197 (bloomFilter=true), to=hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/c44270c3412c22e2c805f466aae6ff56/.tmp/C/1fb73c4dcf674ed099e6e97c11f99372 2023-04-15 02:53:42,072 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/c44270c3412c22e2c805f466aae6ff56/.tmp/A/738be694a3d34996a54af27f4b8b76f0 as hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/c44270c3412c22e2c805f466aae6ff56/A/738be694a3d34996a54af27f4b8b76f0 2023-04-15 02:53:42,078 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/c44270c3412c22e2c805f466aae6ff56/A/738be694a3d34996a54af27f4b8b76f0, entries=150, sequenceid=197, filesize=11.8 K 2023-04-15 02:53:42,080 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/c44270c3412c22e2c805f466aae6ff56/.tmp/B/e91a0eec44524aaea30d1409f1d7aac8 as hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/c44270c3412c22e2c805f466aae6ff56/B/e91a0eec44524aaea30d1409f1d7aac8 2023-04-15 02:53:42,084 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-15 02:53:42,086 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-15 02:53:42,092 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-15 02:53:42,092 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/c44270c3412c22e2c805f466aae6ff56/B/e91a0eec44524aaea30d1409f1d7aac8, entries=150, sequenceid=197, filesize=11.8 K 2023-04-15 02:53:42,093 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-15 02:53:42,093 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/c44270c3412c22e2c805f466aae6ff56/.tmp/C/1fb73c4dcf674ed099e6e97c11f99372 as hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/c44270c3412c22e2c805f466aae6ff56/C/1fb73c4dcf674ed099e6e97c11f99372 2023-04-15 02:53:42,093 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-15 02:53:42,093 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-15 02:53:42,094 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-15 02:53:42,094 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-15 02:53:42,094 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-15 02:53:42,094 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:53:42,094 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:53:42,095 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-15 02:53:42,095 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:42,095 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-15 02:53:42,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-15 02:53:42,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:53:42,095 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:42,095 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:53:42,095 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-15 02:53:42,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-15 02:53:42,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:53:42,096 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-15 02:53:42,096 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-15 02:53:42,096 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-15 02:53:42,096 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-15 02:53:42,096 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-15 02:53:42,096 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-15 02:53:42,096 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-15 02:53:42,096 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-15 02:53:42,096 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681527213260.c44270c3412c22e2c805f466aae6ff56. 2023-04-15 02:53:42,096 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681527213260.c44270c3412c22e2c805f466aae6ff56. started... 2023-04-15 02:53:42,097 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681527213260.c44270c3412c22e2c805f466aae6ff56. as already flushing 2023-04-15 02:53:42,097 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681527213260.c44270c3412c22e2c805f466aae6ff56. 2023-04-15 02:53:42,097 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-15 02:53:42,097 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-15 02:53:42,097 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:42,097 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-15 02:53:42,097 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34681,1681527209133' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-15 02:53:42,098 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:42,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:42,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:53:42,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:53:42,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:53:42,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:53:42,100 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/c44270c3412c22e2c805f466aae6ff56/C/1fb73c4dcf674ed099e6e97c11f99372, entries=150, sequenceid=197, filesize=11.8 K 2023-04-15 02:53:42,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:53:42,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:42,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:53:42,101 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=134.18 KB/137400 for c44270c3412c22e2c805f466aae6ff56 in 917ms, sequenceid=197, compaction requested=true 2023-04-15 02:53:42,101 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for c44270c3412c22e2c805f466aae6ff56: 2023-04-15 02:53:42,101 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34681,1681527209133' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-15 02:53:42,101 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@72d483e8[Count = 0] remaining members to acquire global barrier 2023-04-15 02:53:42,101 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-15 02:53:42,101 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-15 02:53:42,101 DEBUG [RS:0;jenkins-hbase20:34681-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-15 02:53:42,101 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:53:42,101 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-15 02:53:42,101 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-15 02:53:42,101 DEBUG [RS:0;jenkins-hbase20:34681-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-15 02:53:42,102 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:42,102 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-15 02:53:42,102 DEBUG [RS:0;jenkins-hbase20:34681-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36416 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-15 02:53:42,102 DEBUG [RS:0;jenkins-hbase20:34681-shortCompactions-0] regionserver.HStore(1912): c44270c3412c22e2c805f466aae6ff56/A is initiating minor compaction (all files) 2023-04-15 02:53:42,102 DEBUG [RS:0;jenkins-hbase20:34681-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36416 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-15 02:53:42,103 INFO [RS:0;jenkins-hbase20:34681-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of c44270c3412c22e2c805f466aae6ff56/A in TestAcidGuarantees,,1681527213260.c44270c3412c22e2c805f466aae6ff56. 2023-04-15 02:53:42,103 DEBUG [RS:0;jenkins-hbase20:34681-longCompactions-0] regionserver.HStore(1912): c44270c3412c22e2c805f466aae6ff56/B is initiating minor compaction (all files) 2023-04-15 02:53:42,103 INFO [RS:0;jenkins-hbase20:34681-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/c44270c3412c22e2c805f466aae6ff56/A/2f990a41b18641609c1dda35b612f63d, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/c44270c3412c22e2c805f466aae6ff56/A/a4e8adecaa744cf8bdc2810ae3aac070, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/c44270c3412c22e2c805f466aae6ff56/A/738be694a3d34996a54af27f4b8b76f0] into tmpdir=hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/c44270c3412c22e2c805f466aae6ff56/.tmp, totalSize=35.6 K 2023-04-15 02:53:42,103 INFO [RS:0;jenkins-hbase20:34681-longCompactions-0] regionserver.HRegion(2259): Starting compaction of c44270c3412c22e2c805f466aae6ff56/B in TestAcidGuarantees,,1681527213260.c44270c3412c22e2c805f466aae6ff56. 2023-04-15 02:53:42,103 INFO [RS:0;jenkins-hbase20:34681-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/c44270c3412c22e2c805f466aae6ff56/B/00e29523c81a4a4b8618ef27d74d4385, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/c44270c3412c22e2c805f466aae6ff56/B/71f84bdaa00940eda909883ab8648894, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/c44270c3412c22e2c805f466aae6ff56/B/e91a0eec44524aaea30d1409f1d7aac8] into tmpdir=hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/c44270c3412c22e2c805f466aae6ff56/.tmp, totalSize=35.6 K 2023-04-15 02:53:42,103 DEBUG [RS:0;jenkins-hbase20:34681-shortCompactions-0] compactions.Compactor(207): Compacting 2f990a41b18641609c1dda35b612f63d, keycount=150, bloomtype=ROW, size=12.1 K, encoding=NONE, compression=NONE, seqNum=163, earliestPutTs=1681527218332 2023-04-15 02:53:42,103 DEBUG [RS:0;jenkins-hbase20:34681-longCompactions-0] compactions.Compactor(207): Compacting 00e29523c81a4a4b8618ef27d74d4385, keycount=150, bloomtype=ROW, size=12.1 K, encoding=NONE, compression=NONE, seqNum=163, earliestPutTs=1681527218332 2023-04-15 02:53:42,103 DEBUG [RS:0;jenkins-hbase20:34681-shortCompactions-0] compactions.Compactor(207): Compacting a4e8adecaa744cf8bdc2810ae3aac070, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=184, earliestPutTs=1681527218986 2023-04-15 02:53:42,104 DEBUG [RS:0;jenkins-hbase20:34681-longCompactions-0] compactions.Compactor(207): Compacting 71f84bdaa00940eda909883ab8648894, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=184, earliestPutTs=1681527218986 2023-04-15 02:53:42,104 DEBUG [RS:0;jenkins-hbase20:34681-shortCompactions-0] compactions.Compactor(207): Compacting 738be694a3d34996a54af27f4b8b76f0, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=197, earliestPutTs=1681527221180 2023-04-15 02:53:42,104 DEBUG [RS:0;jenkins-hbase20:34681-longCompactions-0] compactions.Compactor(207): Compacting e91a0eec44524aaea30d1409f1d7aac8, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=197, earliestPutTs=1681527221180 2023-04-15 02:53:42,106 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:53:42,106 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] zookeeper.ZKUtil(162): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:53:42,106 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:53:42,107 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-15 02:53:42,107 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-15 02:53:42,107 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-15 02:53:42,107 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34681,1681527209133' in zk 2023-04-15 02:53:42,114 INFO [RS:0;jenkins-hbase20:34681-shortCompactions-0] throttle.PressureAwareThroughputController(145): c44270c3412c22e2c805f466aae6ff56#A#compaction#36 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-15 02:53:42,115 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-15 02:53:42,115 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-15 02:53:42,115 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:42,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:42,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:53:42,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:53:42,115 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-15 02:53:42,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:53:42,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:53:42,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:53:42,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:42,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:53:42,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:53:42,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:42,118 INFO [RS:0;jenkins-hbase20:34681-longCompactions-0] throttle.PressureAwareThroughputController(145): c44270c3412c22e2c805f466aae6ff56#B#compaction#37 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-15 02:53:42,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34681,1681527209133': 2023-04-15 02:53:42,118 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34681,1681527209133' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-15 02:53:42,119 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-15 02:53:42,119 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-15 02:53:42,119 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-15 02:53:42,119 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-15 02:53:42,119 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-15 02:53:42,123 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:53:42,123 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:53:42,123 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-15 02:53:42,124 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:53:42,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:53:42,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:53:42,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:53:42,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:53:42,124 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-15 02:53:42,124 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:42,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:53:42,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-15 02:53:42,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:53:42,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:53:42,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:53:42,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:53:42,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:42,126 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:42,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:53:42,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:53:42,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:42,132 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:42,132 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-15 02:53:42,132 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:53:42,132 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-15 02:53:42,132 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-15 02:53:42,132 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:53:42,132 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-15 02:53:42,132 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-15 02:53:42,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-15 02:53:42,132 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:53:42,132 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-15 02:53:42,132 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:42,132 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:53:42,132 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:53:42,132 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:53:42,132 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-15 02:53:42,132 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-15 02:53:42,133 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-15 02:53:42,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-15 02:53:42,146 DEBUG [RS:0;jenkins-hbase20:34681-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/c44270c3412c22e2c805f466aae6ff56/.tmp/B/033565ea1a184d06b070bb8636f464a2 as hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/c44270c3412c22e2c805f466aae6ff56/B/033565ea1a184d06b070bb8636f464a2 2023-04-15 02:53:42,155 INFO [RS:0;jenkins-hbase20:34681-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in c44270c3412c22e2c805f466aae6ff56/B of c44270c3412c22e2c805f466aae6ff56 into 033565ea1a184d06b070bb8636f464a2(size=12.2 K), total size for store is 12.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-15 02:53:42,155 DEBUG [RS:0;jenkins-hbase20:34681-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/c44270c3412c22e2c805f466aae6ff56/.tmp/A/8f7b03202b8040ad85cab6a1a5522b63 as hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/c44270c3412c22e2c805f466aae6ff56/A/8f7b03202b8040ad85cab6a1a5522b63 2023-04-15 02:53:42,155 DEBUG [RS:0;jenkins-hbase20:34681-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for c44270c3412c22e2c805f466aae6ff56: 2023-04-15 02:53:42,155 INFO [RS:0;jenkins-hbase20:34681-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681527213260.c44270c3412c22e2c805f466aae6ff56., storeName=c44270c3412c22e2c805f466aae6ff56/B, priority=13, startTime=1681527222101; duration=0sec 2023-04-15 02:53:42,155 DEBUG [RS:0;jenkins-hbase20:34681-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-15 02:53:42,155 DEBUG [RS:0;jenkins-hbase20:34681-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-15 02:53:42,156 DEBUG [RS:0;jenkins-hbase20:34681-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36416 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-15 02:53:42,157 DEBUG [RS:0;jenkins-hbase20:34681-longCompactions-0] regionserver.HStore(1912): c44270c3412c22e2c805f466aae6ff56/C is initiating minor compaction (all files) 2023-04-15 02:53:42,157 INFO [RS:0;jenkins-hbase20:34681-longCompactions-0] regionserver.HRegion(2259): Starting compaction of c44270c3412c22e2c805f466aae6ff56/C in TestAcidGuarantees,,1681527213260.c44270c3412c22e2c805f466aae6ff56. 2023-04-15 02:53:42,157 INFO [RS:0;jenkins-hbase20:34681-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/c44270c3412c22e2c805f466aae6ff56/C/c3ee48d87cbb43b6b0cc49b0a53b533f, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/c44270c3412c22e2c805f466aae6ff56/C/845decfbcaff4e56a8eb1e0a5db76d11, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/c44270c3412c22e2c805f466aae6ff56/C/1fb73c4dcf674ed099e6e97c11f99372] into tmpdir=hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/c44270c3412c22e2c805f466aae6ff56/.tmp, totalSize=35.6 K 2023-04-15 02:53:42,158 DEBUG [RS:0;jenkins-hbase20:34681-longCompactions-0] compactions.Compactor(207): Compacting c3ee48d87cbb43b6b0cc49b0a53b533f, keycount=150, bloomtype=ROW, size=12.1 K, encoding=NONE, compression=NONE, seqNum=163, earliestPutTs=1681527218332 2023-04-15 02:53:42,158 DEBUG [RS:0;jenkins-hbase20:34681-longCompactions-0] compactions.Compactor(207): Compacting 845decfbcaff4e56a8eb1e0a5db76d11, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=184, earliestPutTs=1681527218986 2023-04-15 02:53:42,159 DEBUG [RS:0;jenkins-hbase20:34681-longCompactions-0] compactions.Compactor(207): Compacting 1fb73c4dcf674ed099e6e97c11f99372, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=197, earliestPutTs=1681527221180 2023-04-15 02:53:42,162 INFO [RS:0;jenkins-hbase20:34681-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in c44270c3412c22e2c805f466aae6ff56/A of c44270c3412c22e2c805f466aae6ff56 into 8f7b03202b8040ad85cab6a1a5522b63(size=12.2 K), total size for store is 12.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-15 02:53:42,162 DEBUG [RS:0;jenkins-hbase20:34681-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for c44270c3412c22e2c805f466aae6ff56: 2023-04-15 02:53:42,162 INFO [RS:0;jenkins-hbase20:34681-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681527213260.c44270c3412c22e2c805f466aae6ff56., storeName=c44270c3412c22e2c805f466aae6ff56/A, priority=13, startTime=1681527222101; duration=0sec 2023-04-15 02:53:42,162 DEBUG [RS:0;jenkins-hbase20:34681-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-15 02:53:42,169 INFO [RS:0;jenkins-hbase20:34681-longCompactions-0] throttle.PressureAwareThroughputController(145): c44270c3412c22e2c805f466aae6ff56#C#compaction#38 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-15 02:53:42,233 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-15 02:53:42,233 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-15 02:53:42,235 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-15 02:53:42,236 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-15 02:53:42,236 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-15 02:53:42,236 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-15 02:53:42,236 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-15 02:53:42,236 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-15 02:53:42,236 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-15 02:53:42,237 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:53:42,237 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:53:42,238 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-15 02:53:42,238 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:42,238 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-15 02:53:42,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-15 02:53:42,238 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:42,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:53:42,238 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-15 02:53:42,239 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:53:42,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-15 02:53:42,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:53:42,239 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-15 02:53:42,239 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-15 02:53:42,239 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-15 02:53:42,239 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-15 02:53:42,239 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-15 02:53:42,240 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-15 02:53:42,242 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-15 02:53:42,242 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681527213260.c44270c3412c22e2c805f466aae6ff56. 2023-04-15 02:53:42,242 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-15 02:53:42,242 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681527213260.c44270c3412c22e2c805f466aae6ff56. started... 2023-04-15 02:53:42,242 INFO [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing c44270c3412c22e2c805f466aae6ff56 3/3 column families, dataSize=134.18 KB heapSize=375.61 KB 2023-04-15 02:53:42,242 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c44270c3412c22e2c805f466aae6ff56, store=A 2023-04-15 02:53:42,243 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-15 02:53:42,243 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c44270c3412c22e2c805f466aae6ff56, store=B 2023-04-15 02:53:42,243 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-15 02:53:42,243 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c44270c3412c22e2c805f466aae6ff56, store=C 2023-04-15 02:53:42,243 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-15 02:53:42,320 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34681] regionserver.HRegion(9158): Flush requested on c44270c3412c22e2c805f466aae6ff56 2023-04-15 02:53:42,320 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681527213260.c44270c3412c22e2c805f466aae6ff56. as already flushing 2023-04-15 02:53:42,344 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c44270c3412c22e2c805f466aae6ff56, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:53:42,345 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34681] ipc.CallRunner(144): callId: 92 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:38530 deadline: 1681527282341, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c44270c3412c22e2c805f466aae6ff56, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:42,354 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c44270c3412c22e2c805f466aae6ff56, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:53:42,354 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c44270c3412c22e2c805f466aae6ff56, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:53:42,354 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34681] ipc.CallRunner(144): callId: 89 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:38542 deadline: 1681527282351, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c44270c3412c22e2c805f466aae6ff56, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:42,354 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34681] ipc.CallRunner(144): callId: 92 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:38520 deadline: 1681527282352, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c44270c3412c22e2c805f466aae6ff56, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:42,354 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c44270c3412c22e2c805f466aae6ff56, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:53:42,355 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34681] ipc.CallRunner(144): callId: 91 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:38504 deadline: 1681527282353, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c44270c3412c22e2c805f466aae6ff56, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:42,447 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c44270c3412c22e2c805f466aae6ff56, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:53:42,447 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34681] ipc.CallRunner(144): callId: 94 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:38530 deadline: 1681527282446, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c44270c3412c22e2c805f466aae6ff56, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:42,585 DEBUG [RS:0;jenkins-hbase20:34681-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/c44270c3412c22e2c805f466aae6ff56/.tmp/C/4cb2f6c7cea1426cb832a4e7203a86bc as hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/c44270c3412c22e2c805f466aae6ff56/C/4cb2f6c7cea1426cb832a4e7203a86bc 2023-04-15 02:53:42,606 INFO [RS:0;jenkins-hbase20:34681-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in c44270c3412c22e2c805f466aae6ff56/C of c44270c3412c22e2c805f466aae6ff56 into 4cb2f6c7cea1426cb832a4e7203a86bc(size=12.2 K), total size for store is 12.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-15 02:53:42,606 DEBUG [RS:0;jenkins-hbase20:34681-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for c44270c3412c22e2c805f466aae6ff56: 2023-04-15 02:53:42,606 INFO [RS:0;jenkins-hbase20:34681-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681527213260.c44270c3412c22e2c805f466aae6ff56., storeName=c44270c3412c22e2c805f466aae6ff56/C, priority=13, startTime=1681527222101; duration=0sec 2023-04-15 02:53:42,606 DEBUG [RS:0;jenkins-hbase20:34681-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-15 02:53:42,653 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c44270c3412c22e2c805f466aae6ff56, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:53:42,653 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34681] ipc.CallRunner(144): callId: 96 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:38530 deadline: 1681527282651, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c44270c3412c22e2c805f466aae6ff56, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:42,678 INFO [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=234 (bloomFilter=true), to=hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/c44270c3412c22e2c805f466aae6ff56/.tmp/A/3fecd097e4dc418788d8e03e486d995f 2023-04-15 02:53:42,711 INFO [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=234 (bloomFilter=true), to=hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/c44270c3412c22e2c805f466aae6ff56/.tmp/B/ee3572e477764367b4360e7da2e9b6b9 2023-04-15 02:53:42,738 INFO [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=234 (bloomFilter=true), to=hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/c44270c3412c22e2c805f466aae6ff56/.tmp/C/ff9445b047e14cd4ace2d569eb140ab6 2023-04-15 02:53:42,747 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/c44270c3412c22e2c805f466aae6ff56/.tmp/A/3fecd097e4dc418788d8e03e486d995f as hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/c44270c3412c22e2c805f466aae6ff56/A/3fecd097e4dc418788d8e03e486d995f 2023-04-15 02:53:42,755 INFO [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/c44270c3412c22e2c805f466aae6ff56/A/3fecd097e4dc418788d8e03e486d995f, entries=150, sequenceid=234, filesize=11.8 K 2023-04-15 02:53:42,756 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/c44270c3412c22e2c805f466aae6ff56/.tmp/B/ee3572e477764367b4360e7da2e9b6b9 as hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/c44270c3412c22e2c805f466aae6ff56/B/ee3572e477764367b4360e7da2e9b6b9 2023-04-15 02:53:42,763 INFO [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/c44270c3412c22e2c805f466aae6ff56/B/ee3572e477764367b4360e7da2e9b6b9, entries=150, sequenceid=234, filesize=11.8 K 2023-04-15 02:53:42,764 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/c44270c3412c22e2c805f466aae6ff56/.tmp/C/ff9445b047e14cd4ace2d569eb140ab6 as hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/c44270c3412c22e2c805f466aae6ff56/C/ff9445b047e14cd4ace2d569eb140ab6 2023-04-15 02:53:42,772 INFO [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/c44270c3412c22e2c805f466aae6ff56/C/ff9445b047e14cd4ace2d569eb140ab6, entries=150, sequenceid=234, filesize=11.8 K 2023-04-15 02:53:42,773 INFO [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~134.18 KB/137400, heapSize ~375.56 KB/384576, currentSize=46.96 KB/48090 for c44270c3412c22e2c805f466aae6ff56 in 531ms, sequenceid=234, compaction requested=false 2023-04-15 02:53:42,773 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for c44270c3412c22e2c805f466aae6ff56: 2023-04-15 02:53:42,773 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681527213260.c44270c3412c22e2c805f466aae6ff56. 2023-04-15 02:53:42,773 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-15 02:53:42,773 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-15 02:53:42,773 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:42,773 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-15 02:53:42,773 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34681,1681527209133' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-15 02:53:42,774 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:42,775 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:53:42,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:42,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:53:42,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:53:42,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:53:42,775 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:53:42,775 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-15 02:53:42,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:53:42,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:53:42,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:42,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:53:42,779 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34681,1681527209133' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-15 02:53:42,779 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@32bf2576[Count = 0] remaining members to acquire global barrier 2023-04-15 02:53:42,779 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-15 02:53:42,779 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:53:42,780 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:53:42,780 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:53:42,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:53:42,780 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-15 02:53:42,780 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-15 02:53:42,780 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34681,1681527209133' in zk 2023-04-15 02:53:42,781 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:42,781 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-15 02:53:42,782 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:42,782 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-15 02:53:42,782 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-15 02:53:42,782 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-15 02:53:42,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:42,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:53:42,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:53:42,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:53:42,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:53:42,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:53:42,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:42,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:53:42,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:53:42,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:42,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34681,1681527209133': 2023-04-15 02:53:42,785 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34681,1681527209133' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-15 02:53:42,785 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-15 02:53:42,785 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-15 02:53:42,785 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-15 02:53:42,785 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-15 02:53:42,785 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-15 02:53:42,787 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:53:42,787 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-15 02:53:42,787 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:53:42,787 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:53:42,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:53:42,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:53:42,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:53:42,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:53:42,788 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-15 02:53:42,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-15 02:53:42,788 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:42,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:53:42,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:53:42,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:53:42,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:53:42,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:53:42,790 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:42,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:42,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:53:42,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:53:42,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:42,794 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-15 02:53:42,794 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-15 02:53:42,794 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-15 02:53:42,794 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-15 02:53:42,794 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:42,794 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:53:42,794 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:53:42,794 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:42,794 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:53:42,794 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:53:42,794 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:53:42,794 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-15 02:53:42,794 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-15 02:53:42,794 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-15 02:53:42,794 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-15 02:53:42,794 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-15 02:53:42,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-15 02:53:42,795 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-15 02:53:42,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-15 02:53:42,894 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-15 02:53:42,895 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-15 02:53:42,897 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-15 02:53:42,898 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-15 02:53:42,898 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-15 02:53:42,899 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-15 02:53:42,899 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-15 02:53:42,899 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-15 02:53:42,899 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-15 02:53:42,899 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:53:42,899 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:53:42,900 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-15 02:53:42,900 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:42,900 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-15 02:53:42,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-15 02:53:42,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:53:42,901 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:42,901 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-15 02:53:42,901 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:53:42,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-15 02:53:42,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:53:42,901 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-15 02:53:42,902 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-15 02:53:42,902 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-15 02:53:42,902 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-15 02:53:42,902 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-15 02:53:42,902 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-15 02:53:42,902 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-15 02:53:42,902 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-15 02:53:42,902 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681527213260.c44270c3412c22e2c805f466aae6ff56. 2023-04-15 02:53:42,903 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681527213260.c44270c3412c22e2c805f466aae6ff56. started... 2023-04-15 02:53:42,903 INFO [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing c44270c3412c22e2c805f466aae6ff56 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-15 02:53:42,904 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c44270c3412c22e2c805f466aae6ff56, store=A 2023-04-15 02:53:42,904 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-15 02:53:42,904 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c44270c3412c22e2c805f466aae6ff56, store=B 2023-04-15 02:53:42,904 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-15 02:53:42,905 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c44270c3412c22e2c805f466aae6ff56, store=C 2023-04-15 02:53:42,905 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-15 02:53:42,938 INFO [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=245 (bloomFilter=true), to=hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/c44270c3412c22e2c805f466aae6ff56/.tmp/A/90066fdb8faf41cd90972274f2085d6c 2023-04-15 02:53:42,963 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681527213260.c44270c3412c22e2c805f466aae6ff56. as already flushing 2023-04-15 02:53:42,963 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34681] regionserver.HRegion(9158): Flush requested on c44270c3412c22e2c805f466aae6ff56 2023-04-15 02:53:43,016 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681527223006, max timestamp=1681527223013Num uniques 859; 2023-04-15 02:53:43,016 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681527223006, max timestamp=1681527223013Num uniques 859; 2023-04-15 02:53:43,016 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681527223006, max timestamp=1681527223013Num uniques 859; 2023-04-15 02:53:43,016 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-5] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-15 02:53:43,016 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-6] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-15 02:53:43,016 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-7] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-15 02:53:43,066 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c44270c3412c22e2c805f466aae6ff56, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:53:43,066 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34681] ipc.CallRunner(144): callId: 132 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:38530 deadline: 1681527283062, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c44270c3412c22e2c805f466aae6ff56, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:43,167 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c44270c3412c22e2c805f466aae6ff56, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:53:43,168 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34681] ipc.CallRunner(144): callId: 134 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:38530 deadline: 1681527283167, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c44270c3412c22e2c805f466aae6ff56, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:43,360 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c44270c3412c22e2c805f466aae6ff56, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:53:43,360 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34681] ipc.CallRunner(144): callId: 94 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:38520 deadline: 1681527283358, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c44270c3412c22e2c805f466aae6ff56, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:43,360 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c44270c3412c22e2c805f466aae6ff56, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:53:43,361 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34681] ipc.CallRunner(144): callId: 91 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:38542 deadline: 1681527283359, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c44270c3412c22e2c805f466aae6ff56, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:43,368 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c44270c3412c22e2c805f466aae6ff56, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:53:43,368 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34681] ipc.CallRunner(144): callId: 93 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:38504 deadline: 1681527283365, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c44270c3412c22e2c805f466aae6ff56, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:43,370 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c44270c3412c22e2c805f466aae6ff56, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:53:43,370 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34681] ipc.CallRunner(144): callId: 136 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:38530 deadline: 1681527283369, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c44270c3412c22e2c805f466aae6ff56, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:43,392 INFO [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=245 (bloomFilter=true), to=hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/c44270c3412c22e2c805f466aae6ff56/.tmp/B/a517dd6eae5d4726a3b1947f94b7de28 2023-04-15 02:53:43,406 INFO [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=245 (bloomFilter=true), to=hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/c44270c3412c22e2c805f466aae6ff56/.tmp/C/935006ea70d34e00b33b6556dfd065fd 2023-04-15 02:53:43,414 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/c44270c3412c22e2c805f466aae6ff56/.tmp/A/90066fdb8faf41cd90972274f2085d6c as hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/c44270c3412c22e2c805f466aae6ff56/A/90066fdb8faf41cd90972274f2085d6c 2023-04-15 02:53:43,421 INFO [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/c44270c3412c22e2c805f466aae6ff56/A/90066fdb8faf41cd90972274f2085d6c, entries=100, sequenceid=245, filesize=9.4 K 2023-04-15 02:53:43,422 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/c44270c3412c22e2c805f466aae6ff56/.tmp/B/a517dd6eae5d4726a3b1947f94b7de28 as hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/c44270c3412c22e2c805f466aae6ff56/B/a517dd6eae5d4726a3b1947f94b7de28 2023-04-15 02:53:43,428 INFO [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/c44270c3412c22e2c805f466aae6ff56/B/a517dd6eae5d4726a3b1947f94b7de28, entries=100, sequenceid=245, filesize=9.4 K 2023-04-15 02:53:43,429 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/c44270c3412c22e2c805f466aae6ff56/.tmp/C/935006ea70d34e00b33b6556dfd065fd as hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/c44270c3412c22e2c805f466aae6ff56/C/935006ea70d34e00b33b6556dfd065fd 2023-04-15 02:53:43,435 INFO [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/c44270c3412c22e2c805f466aae6ff56/C/935006ea70d34e00b33b6556dfd065fd, entries=100, sequenceid=245, filesize=9.4 K 2023-04-15 02:53:43,436 INFO [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=134.18 KB/137400 for c44270c3412c22e2c805f466aae6ff56 in 533ms, sequenceid=245, compaction requested=true 2023-04-15 02:53:43,436 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for c44270c3412c22e2c805f466aae6ff56: 2023-04-15 02:53:43,436 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681527213260.c44270c3412c22e2c805f466aae6ff56. 2023-04-15 02:53:43,436 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-15 02:53:43,436 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-15 02:53:43,436 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:43,436 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-15 02:53:43,436 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34681,1681527209133' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-15 02:53:43,437 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:53:43,437 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:53:43,437 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-15 02:53:43,439 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:43,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:43,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:53:43,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:53:43,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:53:43,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:53:43,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:53:43,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:43,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:53:43,442 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34681,1681527209133' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-15 02:53:43,442 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@46aa3cd4[Count = 0] remaining members to acquire global barrier 2023-04-15 02:53:43,442 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-15 02:53:43,442 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:53:43,443 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:53:43,443 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:53:43,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:53:43,443 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-15 02:53:43,443 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-15 02:53:43,443 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:43,443 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-15 02:53:43,443 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34681,1681527209133' in zk 2023-04-15 02:53:43,444 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:43,444 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-15 02:53:43,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:43,444 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-15 02:53:43,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:53:43,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:53:43,444 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-15 02:53:43,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:53:43,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:53:43,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:53:43,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:43,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:53:43,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:53:43,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:43,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34681,1681527209133': 2023-04-15 02:53:43,446 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34681,1681527209133' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-15 02:53:43,446 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-15 02:53:43,446 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-15 02:53:43,446 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-15 02:53:43,446 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-15 02:53:43,446 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-15 02:53:43,448 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:53:43,448 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:53:43,448 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:53:43,448 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-15 02:53:43,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:53:43,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:53:43,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:53:43,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:53:43,448 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-15 02:53:43,448 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:43,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-15 02:53:43,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:53:43,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:53:43,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:53:43,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:53:43,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:53:43,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:43,450 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:43,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:53:43,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:53:43,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:43,452 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-15 02:53:43,452 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-15 02:53:43,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-15 02:53:43,452 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:43,452 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-15 02:53:43,452 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:53:43,452 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:53:43,452 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-15 02:53:43,452 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-15 02:53:43,452 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-15 02:53:43,452 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:43,452 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:53:43,452 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-15 02:53:43,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-15 02:53:43,452 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-15 02:53:43,452 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:53:43,452 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-15 02:53:43,452 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:53:43,553 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-15 02:53:43,553 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-15 02:53:43,555 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-15 02:53:43,556 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-15 02:53:43,556 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-15 02:53:43,556 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-15 02:53:43,556 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-15 02:53:43,556 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-15 02:53:43,556 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-15 02:53:43,556 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:53:43,556 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:53:43,557 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-15 02:53:43,557 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:43,557 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-15 02:53:43,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-15 02:53:43,558 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:43,558 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-15 02:53:43,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:53:43,558 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:53:43,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-15 02:53:43,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:53:43,558 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-15 02:53:43,558 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-15 02:53:43,558 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-15 02:53:43,559 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-15 02:53:43,559 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-15 02:53:43,559 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-15 02:53:43,559 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-15 02:53:43,559 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-15 02:53:43,559 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681527213260.c44270c3412c22e2c805f466aae6ff56. 2023-04-15 02:53:43,559 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681527213260.c44270c3412c22e2c805f466aae6ff56. started... 2023-04-15 02:53:43,559 INFO [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing c44270c3412c22e2c805f466aae6ff56 3/3 column families, dataSize=134.18 KB heapSize=384.98 KB 2023-04-15 02:53:43,559 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c44270c3412c22e2c805f466aae6ff56, store=A 2023-04-15 02:53:43,559 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-15 02:53:43,559 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c44270c3412c22e2c805f466aae6ff56, store=B 2023-04-15 02:53:43,559 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-15 02:53:43,559 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c44270c3412c22e2c805f466aae6ff56, store=C 2023-04-15 02:53:43,559 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-15 02:53:43,675 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34681] regionserver.HRegion(9158): Flush requested on c44270c3412c22e2c805f466aae6ff56 2023-04-15 02:53:43,675 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681527213260.c44270c3412c22e2c805f466aae6ff56. as already flushing 2023-04-15 02:53:43,698 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c44270c3412c22e2c805f466aae6ff56, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:53:43,698 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34681] ipc.CallRunner(144): callId: 145 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:38530 deadline: 1681527283696, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c44270c3412c22e2c805f466aae6ff56, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:43,805 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c44270c3412c22e2c805f466aae6ff56, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:53:43,806 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34681] ipc.CallRunner(144): callId: 147 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:38530 deadline: 1681527283801, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c44270c3412c22e2c805f466aae6ff56, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:43,972 INFO [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=282 (bloomFilter=true), to=hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/c44270c3412c22e2c805f466aae6ff56/.tmp/A/ae40346c39e94bcdb9eb81fbc0a744f2 2023-04-15 02:53:43,988 INFO [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=282 (bloomFilter=true), to=hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/c44270c3412c22e2c805f466aae6ff56/.tmp/B/6e8954bf57a44d1598c74e296874d020 2023-04-15 02:53:44,011 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c44270c3412c22e2c805f466aae6ff56, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:53:44,011 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34681] ipc.CallRunner(144): callId: 149 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:38530 deadline: 1681527284008, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c44270c3412c22e2c805f466aae6ff56, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:44,013 INFO [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=282 (bloomFilter=true), to=hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/c44270c3412c22e2c805f466aae6ff56/.tmp/C/c4954ab54de54450a33fc1030524f6ed 2023-04-15 02:53:44,020 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/c44270c3412c22e2c805f466aae6ff56/.tmp/A/ae40346c39e94bcdb9eb81fbc0a744f2 as hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/c44270c3412c22e2c805f466aae6ff56/A/ae40346c39e94bcdb9eb81fbc0a744f2 2023-04-15 02:53:44,025 INFO [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/c44270c3412c22e2c805f466aae6ff56/A/ae40346c39e94bcdb9eb81fbc0a744f2, entries=150, sequenceid=282, filesize=11.9 K 2023-04-15 02:53:44,026 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/c44270c3412c22e2c805f466aae6ff56/.tmp/B/6e8954bf57a44d1598c74e296874d020 as hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/c44270c3412c22e2c805f466aae6ff56/B/6e8954bf57a44d1598c74e296874d020 2023-04-15 02:53:44,033 INFO [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/c44270c3412c22e2c805f466aae6ff56/B/6e8954bf57a44d1598c74e296874d020, entries=150, sequenceid=282, filesize=11.9 K 2023-04-15 02:53:44,034 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/c44270c3412c22e2c805f466aae6ff56/.tmp/C/c4954ab54de54450a33fc1030524f6ed as hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/c44270c3412c22e2c805f466aae6ff56/C/c4954ab54de54450a33fc1030524f6ed 2023-04-15 02:53:44,041 INFO [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/c44270c3412c22e2c805f466aae6ff56/C/c4954ab54de54450a33fc1030524f6ed, entries=150, sequenceid=282, filesize=11.9 K 2023-04-15 02:53:44,042 INFO [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~134.18 KB/137400, heapSize ~384.94 KB/394176, currentSize=46.96 KB/48090 for c44270c3412c22e2c805f466aae6ff56 in 483ms, sequenceid=282, compaction requested=true 2023-04-15 02:53:44,042 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for c44270c3412c22e2c805f466aae6ff56: 2023-04-15 02:53:44,042 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681527213260.c44270c3412c22e2c805f466aae6ff56. 2023-04-15 02:53:44,042 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-15 02:53:44,042 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-15 02:53:44,042 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:44,042 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-15 02:53:44,042 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34681,1681527209133' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-15 02:53:44,044 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:44,044 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:53:44,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:44,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:53:44,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:53:44,044 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:53:44,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:53:44,044 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-15 02:53:44,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:53:44,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:53:44,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:44,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:53:44,047 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34681,1681527209133' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-15 02:53:44,047 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7074ad8d[Count = 0] remaining members to acquire global barrier 2023-04-15 02:53:44,047 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-15 02:53:44,047 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:53:44,048 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:53:44,048 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:53:44,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:53:44,048 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-15 02:53:44,048 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-15 02:53:44,048 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34681,1681527209133' in zk 2023-04-15 02:53:44,048 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:44,048 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-15 02:53:44,049 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-15 02:53:44,049 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-15 02:53:44,049 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:44,049 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-15 02:53:44,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:44,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:53:44,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:53:44,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:53:44,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:53:44,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:53:44,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:44,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:53:44,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:53:44,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:44,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34681,1681527209133': 2023-04-15 02:53:44,055 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34681,1681527209133' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-15 02:53:44,055 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-15 02:53:44,055 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-15 02:53:44,055 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-15 02:53:44,055 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-15 02:53:44,055 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-15 02:53:44,061 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:53:44,061 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:53:44,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:53:44,061 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:53:44,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:53:44,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:53:44,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:53:44,061 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-15 02:53:44,062 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-15 02:53:44,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-15 02:53:44,062 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:44,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:53:44,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:53:44,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:53:44,063 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:44,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:53:44,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:53:44,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:44,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:53:44,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:53:44,069 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-15 02:53:44,069 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-15 02:53:44,069 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-15 02:53:44,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-15 02:53:44,070 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:44,070 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:53:44,070 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:53:44,070 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-15 02:53:44,070 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:44,070 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:53:44,070 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:53:44,070 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:53:44,070 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-15 02:53:44,070 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-15 02:53:44,070 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-15 02:53:44,071 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-15 02:53:44,071 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-15 02:53:44,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-15 02:53:44,071 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-15 02:53:44,171 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-15 02:53:44,171 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-15 02:53:44,173 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-15 02:53:44,174 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-15 02:53:44,174 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-15 02:53:44,174 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-15 02:53:44,174 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-15 02:53:44,174 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-15 02:53:44,174 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-15 02:53:44,174 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:53:44,174 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:53:44,175 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-15 02:53:44,175 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:44,175 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-15 02:53:44,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-15 02:53:44,176 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:44,176 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-15 02:53:44,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:53:44,176 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:53:44,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-15 02:53:44,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:53:44,176 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-15 02:53:44,177 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-15 02:53:44,177 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-15 02:53:44,177 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-15 02:53:44,177 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-15 02:53:44,177 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-15 02:53:44,190 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-15 02:53:44,190 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-15 02:53:44,194 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681527213260.c44270c3412c22e2c805f466aae6ff56. 2023-04-15 02:53:44,194 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681527213260.c44270c3412c22e2c805f466aae6ff56. started... 2023-04-15 02:53:44,194 INFO [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing c44270c3412c22e2c805f466aae6ff56 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-15 02:53:44,194 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c44270c3412c22e2c805f466aae6ff56, store=A 2023-04-15 02:53:44,195 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-15 02:53:44,195 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c44270c3412c22e2c805f466aae6ff56, store=B 2023-04-15 02:53:44,195 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-15 02:53:44,195 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c44270c3412c22e2c805f466aae6ff56, store=C 2023-04-15 02:53:44,195 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-15 02:53:44,215 INFO [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=292 (bloomFilter=true), to=hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/c44270c3412c22e2c805f466aae6ff56/.tmp/A/c0f7d1d451b94af2b158f512a2c089ed 2023-04-15 02:53:44,230 INFO [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=292 (bloomFilter=true), to=hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/c44270c3412c22e2c805f466aae6ff56/.tmp/B/575038b7953e422695ac1f08a68ff389 2023-04-15 02:53:44,243 INFO [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=292 (bloomFilter=true), to=hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/c44270c3412c22e2c805f466aae6ff56/.tmp/C/d10899e00fe64a74810f1c8fe7034cc5 2023-04-15 02:53:44,250 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/c44270c3412c22e2c805f466aae6ff56/.tmp/A/c0f7d1d451b94af2b158f512a2c089ed as hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/c44270c3412c22e2c805f466aae6ff56/A/c0f7d1d451b94af2b158f512a2c089ed 2023-04-15 02:53:44,256 INFO [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/c44270c3412c22e2c805f466aae6ff56/A/c0f7d1d451b94af2b158f512a2c089ed, entries=150, sequenceid=292, filesize=11.9 K 2023-04-15 02:53:44,258 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/c44270c3412c22e2c805f466aae6ff56/.tmp/B/575038b7953e422695ac1f08a68ff389 as hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/c44270c3412c22e2c805f466aae6ff56/B/575038b7953e422695ac1f08a68ff389 2023-04-15 02:53:44,263 INFO [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/c44270c3412c22e2c805f466aae6ff56/B/575038b7953e422695ac1f08a68ff389, entries=150, sequenceid=292, filesize=11.9 K 2023-04-15 02:53:44,265 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/c44270c3412c22e2c805f466aae6ff56/.tmp/C/d10899e00fe64a74810f1c8fe7034cc5 as hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/c44270c3412c22e2c805f466aae6ff56/C/d10899e00fe64a74810f1c8fe7034cc5 2023-04-15 02:53:44,270 INFO [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/c44270c3412c22e2c805f466aae6ff56/C/d10899e00fe64a74810f1c8fe7034cc5, entries=150, sequenceid=292, filesize=11.9 K 2023-04-15 02:53:44,271 INFO [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=0 B/0 for c44270c3412c22e2c805f466aae6ff56 in 77ms, sequenceid=292, compaction requested=true 2023-04-15 02:53:44,271 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for c44270c3412c22e2c805f466aae6ff56: 2023-04-15 02:53:44,271 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681527213260.c44270c3412c22e2c805f466aae6ff56. 2023-04-15 02:53:44,271 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-15 02:53:44,272 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-15 02:53:44,272 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:44,272 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-15 02:53:44,272 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34681,1681527209133' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-15 02:53:44,273 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:44,273 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:53:44,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:44,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:53:44,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:53:44,273 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:53:44,273 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-15 02:53:44,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:53:44,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:53:44,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:53:44,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:44,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:53:44,275 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34681,1681527209133' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-15 02:53:44,275 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@26e9aba8[Count = 0] remaining members to acquire global barrier 2023-04-15 02:53:44,275 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-15 02:53:44,275 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:53:44,275 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:53:44,276 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:53:44,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:53:44,276 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-15 02:53:44,276 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-15 02:53:44,276 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34681,1681527209133' in zk 2023-04-15 02:53:44,276 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:44,276 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-15 02:53:44,277 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:44,277 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-15 02:53:44,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:44,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:53:44,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:53:44,277 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-15 02:53:44,277 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-15 02:53:44,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:53:44,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:53:44,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:53:44,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:44,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:53:44,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:53:44,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:44,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34681,1681527209133': 2023-04-15 02:53:44,279 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34681,1681527209133' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-15 02:53:44,279 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-15 02:53:44,280 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-15 02:53:44,280 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-15 02:53:44,280 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-15 02:53:44,280 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-15 02:53:44,281 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:53:44,281 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-15 02:53:44,281 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:53:44,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:53:44,281 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:53:44,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:53:44,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:53:44,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:53:44,281 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-15 02:53:44,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-15 02:53:44,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:53:44,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:53:44,281 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:44,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:53:44,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:53:44,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:53:44,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:44,282 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:44,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:53:44,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:53:44,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:44,284 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-15 02:53:44,284 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-15 02:53:44,284 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:44,284 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-15 02:53:44,284 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:53:44,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-15 02:53:44,285 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-15 02:53:44,285 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-15 02:53:44,285 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-15 02:53:44,285 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:53:44,285 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:44,285 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-15 02:53:44,285 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-15 02:53:44,285 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:53:44,285 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:53:44,285 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-15 02:53:44,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-15 02:53:44,285 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:53:44,327 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34681] regionserver.HRegion(9158): Flush requested on c44270c3412c22e2c805f466aae6ff56 2023-04-15 02:53:44,327 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing c44270c3412c22e2c805f466aae6ff56 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-15 02:53:44,328 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c44270c3412c22e2c805f466aae6ff56, store=A 2023-04-15 02:53:44,328 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-15 02:53:44,328 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c44270c3412c22e2c805f466aae6ff56, store=B 2023-04-15 02:53:44,328 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-15 02:53:44,328 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c44270c3412c22e2c805f466aae6ff56, store=C 2023-04-15 02:53:44,328 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-15 02:53:44,385 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-15 02:53:44,386 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-15 02:53:44,387 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-15 02:53:44,388 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-15 02:53:44,388 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-15 02:53:44,388 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-15 02:53:44,388 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-15 02:53:44,388 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-15 02:53:44,388 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-15 02:53:44,389 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:53:44,389 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:53:44,389 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:44,389 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-15 02:53:44,390 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-15 02:53:44,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-15 02:53:44,390 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:44,390 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-15 02:53:44,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:53:44,391 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681527224366, max timestamp=1681527224389Num uniques 859; 2023-04-15 02:53:44,391 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-2] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-15 02:53:44,391 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681527224366, max timestamp=1681527224389Num uniques 859; 2023-04-15 02:53:44,391 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-1] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-15 02:53:44,392 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:53:44,392 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681527224366, max timestamp=1681527224389Num uniques 859; 2023-04-15 02:53:44,392 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-8] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-15 02:53:44,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-15 02:53:44,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:53:44,393 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-15 02:53:44,393 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-15 02:53:44,393 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-15 02:53:44,393 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-15 02:53:44,393 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-15 02:53:44,393 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-15 02:53:44,393 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-15 02:53:44,393 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-15 02:53:44,393 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681527213260.c44270c3412c22e2c805f466aae6ff56. 2023-04-15 02:53:44,394 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681527213260.c44270c3412c22e2c805f466aae6ff56. started... 2023-04-15 02:53:44,394 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681527213260.c44270c3412c22e2c805f466aae6ff56. as already flushing 2023-04-15 02:53:44,394 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681527213260.c44270c3412c22e2c805f466aae6ff56. 2023-04-15 02:53:44,394 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-15 02:53:44,394 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-15 02:53:44,394 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:44,394 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-15 02:53:44,394 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34681,1681527209133' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-15 02:53:44,395 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:44,395 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:53:44,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:44,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:53:44,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:53:44,395 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:53:44,396 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-15 02:53:44,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:53:44,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:53:44,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:53:44,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:44,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:53:44,397 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34681,1681527209133' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-15 02:53:44,397 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6d36ae96[Count = 0] remaining members to acquire global barrier 2023-04-15 02:53:44,397 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-15 02:53:44,397 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:53:44,397 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:53:44,397 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:53:44,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:53:44,398 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-15 02:53:44,398 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:44,398 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-15 02:53:44,398 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-15 02:53:44,398 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34681,1681527209133' in zk 2023-04-15 02:53:44,399 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:44,399 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-15 02:53:44,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:44,399 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-15 02:53:44,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:53:44,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:53:44,399 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-15 02:53:44,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:53:44,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:53:44,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:53:44,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:44,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:53:44,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:53:44,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:44,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34681,1681527209133': 2023-04-15 02:53:44,402 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34681,1681527209133' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-15 02:53:44,402 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-15 02:53:44,402 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-15 02:53:44,402 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-15 02:53:44,402 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-15 02:53:44,402 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-15 02:53:44,403 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:53:44,403 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:53:44,403 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:53:44,403 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-15 02:53:44,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:53:44,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:53:44,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:53:44,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:53:44,403 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:44,403 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-15 02:53:44,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-15 02:53:44,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:53:44,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:53:44,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:53:44,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:53:44,404 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:44,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:53:44,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:44,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:53:44,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:53:44,407 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:44,407 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:53:44,407 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:53:44,407 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:44,407 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-15 02:53:44,407 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:53:44,407 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:53:44,407 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:53:44,407 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-15 02:53:44,407 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-15 02:53:44,407 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-15 02:53:44,407 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-15 02:53:44,407 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-15 02:53:44,407 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-15 02:53:44,408 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-15 02:53:44,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-15 02:53:44,408 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-15 02:53:44,408 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-15 02:53:44,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-15 02:53:44,471 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c44270c3412c22e2c805f466aae6ff56, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:53:44,471 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34681] ipc.CallRunner(144): callId: 192 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:38530 deadline: 1681527284470, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c44270c3412c22e2c805f466aae6ff56, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:44,508 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-15 02:53:44,509 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-15 02:53:44,510 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-15 02:53:44,511 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-15 02:53:44,511 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-15 02:53:44,511 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-15 02:53:44,511 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-15 02:53:44,511 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-15 02:53:44,511 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-15 02:53:44,512 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:53:44,512 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:53:44,513 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:44,513 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-15 02:53:44,513 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-15 02:53:44,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-15 02:53:44,513 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:44,513 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-15 02:53:44,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:53:44,513 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:53:44,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-15 02:53:44,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:53:44,514 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-15 02:53:44,514 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-15 02:53:44,514 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-15 02:53:44,514 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-15 02:53:44,514 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-15 02:53:44,514 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-15 02:53:44,518 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-15 02:53:44,518 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681527213260.c44270c3412c22e2c805f466aae6ff56. 2023-04-15 02:53:44,518 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-15 02:53:44,518 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681527213260.c44270c3412c22e2c805f466aae6ff56. started... 2023-04-15 02:53:44,518 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681527213260.c44270c3412c22e2c805f466aae6ff56. as already flushing 2023-04-15 02:53:44,518 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681527213260.c44270c3412c22e2c805f466aae6ff56. 2023-04-15 02:53:44,518 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-15 02:53:44,518 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-15 02:53:44,518 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:44,518 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-15 02:53:44,519 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34681,1681527209133' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-15 02:53:44,532 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:44,532 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:53:44,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:44,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:53:44,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:53:44,532 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:53:44,532 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-15 02:53:44,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:53:44,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:53:44,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:53:44,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:44,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:53:44,534 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34681,1681527209133' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-15 02:53:44,534 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7a6d7378[Count = 0] remaining members to acquire global barrier 2023-04-15 02:53:44,534 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-15 02:53:44,535 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:53:44,535 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:53:44,535 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:53:44,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:53:44,536 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-15 02:53:44,536 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-15 02:53:44,536 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34681,1681527209133' in zk 2023-04-15 02:53:44,536 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:44,536 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-15 02:53:44,536 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:44,536 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-15 02:53:44,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:44,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:53:44,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:53:44,536 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-15 02:53:44,537 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-15 02:53:44,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:53:44,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:53:44,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:53:44,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:44,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:53:44,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:53:44,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:44,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34681,1681527209133': 2023-04-15 02:53:44,538 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34681,1681527209133' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-15 02:53:44,538 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-15 02:53:44,539 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-15 02:53:44,539 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-15 02:53:44,539 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-15 02:53:44,539 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-15 02:53:44,540 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:53:44,540 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:53:44,540 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-15 02:53:44,540 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:53:44,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:53:44,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:53:44,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:53:44,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:53:44,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:53:44,541 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-15 02:53:44,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-15 02:53:44,541 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:44,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:53:44,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:53:44,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:53:44,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:53:44,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:44,542 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:44,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:53:44,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:53:44,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:44,544 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:44,544 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-15 02:53:44,544 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-15 02:53:44,544 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-15 02:53:44,544 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:53:44,545 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-15 02:53:44,545 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-15 02:53:44,545 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-15 02:53:44,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-15 02:53:44,545 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-15 02:53:44,545 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-15 02:53:44,545 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:53:44,545 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-15 02:53:44,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-15 02:53:44,545 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:44,545 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:53:44,545 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:53:44,545 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:53:44,574 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c44270c3412c22e2c805f466aae6ff56, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:53:44,574 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34681] ipc.CallRunner(144): callId: 194 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:38530 deadline: 1681527284572, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c44270c3412c22e2c805f466aae6ff56, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:44,645 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-15 02:53:44,646 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-15 02:53:44,647 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-15 02:53:44,648 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-15 02:53:44,648 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-15 02:53:44,648 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-15 02:53:44,648 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-15 02:53:44,648 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-15 02:53:44,648 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-15 02:53:44,649 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:53:44,649 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:53:44,650 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:44,650 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-15 02:53:44,650 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-15 02:53:44,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-15 02:53:44,650 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:44,650 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-15 02:53:44,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:53:44,650 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:53:44,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-15 02:53:44,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:53:44,651 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-15 02:53:44,651 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-15 02:53:44,651 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-15 02:53:44,651 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-15 02:53:44,651 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-15 02:53:44,651 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-15 02:53:44,651 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-15 02:53:44,651 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-15 02:53:44,651 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681527213260.c44270c3412c22e2c805f466aae6ff56. 2023-04-15 02:53:44,651 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681527213260.c44270c3412c22e2c805f466aae6ff56. started... 2023-04-15 02:53:44,651 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681527213260.c44270c3412c22e2c805f466aae6ff56. as already flushing 2023-04-15 02:53:44,651 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681527213260.c44270c3412c22e2c805f466aae6ff56. 2023-04-15 02:53:44,651 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-15 02:53:44,651 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-15 02:53:44,652 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:44,652 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-15 02:53:44,652 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34681,1681527209133' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-15 02:53:44,653 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:44,653 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:53:44,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:44,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:53:44,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:53:44,653 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:53:44,653 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-15 02:53:44,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:53:44,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:53:44,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:53:44,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:44,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:53:44,654 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34681,1681527209133' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-15 02:53:44,654 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2fa0a17d[Count = 0] remaining members to acquire global barrier 2023-04-15 02:53:44,654 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-15 02:53:44,654 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:53:44,654 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:53:44,655 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:53:44,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:53:44,655 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-15 02:53:44,655 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-15 02:53:44,655 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34681,1681527209133' in zk 2023-04-15 02:53:44,655 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:44,655 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-15 02:53:44,655 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:44,655 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-15 02:53:44,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:44,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:53:44,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:53:44,655 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-15 02:53:44,656 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-15 02:53:44,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:53:44,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:53:44,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:53:44,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:44,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:53:44,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:53:44,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:44,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34681,1681527209133': 2023-04-15 02:53:44,657 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34681,1681527209133' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-15 02:53:44,657 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-15 02:53:44,657 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-15 02:53:44,657 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-15 02:53:44,657 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-15 02:53:44,657 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-15 02:53:44,658 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:53:44,658 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:44,658 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:53:44,658 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:53:44,658 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-15 02:53:44,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:53:44,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:53:44,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:53:44,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:53:44,659 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-15 02:53:44,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-15 02:53:44,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:53:44,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:53:44,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:53:44,659 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:44,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:53:44,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:53:44,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:44,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:53:44,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:53:44,661 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:44,661 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:53:44,661 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-15 02:53:44,661 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:53:44,661 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-15 02:53:44,661 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-15 02:53:44,661 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-15 02:53:44,661 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:44,662 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:53:44,662 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:53:44,661 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-15 02:53:44,661 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-15 02:53:44,662 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-15 02:53:44,662 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-15 02:53:44,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-15 02:53:44,662 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:53:44,662 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-15 02:53:44,662 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-15 02:53:44,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-15 02:53:44,737 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=302 (bloomFilter=true), to=hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/c44270c3412c22e2c805f466aae6ff56/.tmp/A/9b3a32d29fc74086b895f2308fd5a598 2023-04-15 02:53:44,762 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-15 02:53:44,762 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-15 02:53:44,764 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-15 02:53:44,764 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-15 02:53:44,765 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-15 02:53:44,765 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-15 02:53:44,765 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-15 02:53:44,765 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-15 02:53:44,765 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-15 02:53:44,765 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:53:44,765 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:53:44,766 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-15 02:53:44,766 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:44,766 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-15 02:53:44,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-15 02:53:44,766 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:44,766 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-15 02:53:44,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:53:44,767 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:53:44,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-15 02:53:44,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:53:44,767 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-15 02:53:44,767 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-15 02:53:44,767 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-15 02:53:44,767 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-15 02:53:44,767 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-15 02:53:44,767 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-15 02:53:44,768 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-15 02:53:44,768 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-15 02:53:44,768 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681527213260.c44270c3412c22e2c805f466aae6ff56. 2023-04-15 02:53:44,768 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681527213260.c44270c3412c22e2c805f466aae6ff56. started... 2023-04-15 02:53:44,768 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681527213260.c44270c3412c22e2c805f466aae6ff56. as already flushing 2023-04-15 02:53:44,768 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681527213260.c44270c3412c22e2c805f466aae6ff56. 2023-04-15 02:53:44,768 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-15 02:53:44,768 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-15 02:53:44,768 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:44,768 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-15 02:53:44,768 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34681,1681527209133' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-15 02:53:44,769 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:44,769 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:53:44,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:44,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:53:44,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:53:44,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:53:44,770 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:53:44,770 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-15 02:53:44,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:53:44,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:53:44,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:44,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:53:44,771 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34681,1681527209133' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-15 02:53:44,771 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@447febca[Count = 0] remaining members to acquire global barrier 2023-04-15 02:53:44,771 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-15 02:53:44,771 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:53:44,771 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:53:44,771 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:53:44,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:53:44,772 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-15 02:53:44,772 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-15 02:53:44,772 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34681,1681527209133' in zk 2023-04-15 02:53:44,772 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:44,772 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-15 02:53:44,772 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:44,772 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-15 02:53:44,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:44,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:53:44,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:53:44,772 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-15 02:53:44,773 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-15 02:53:44,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:53:44,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:53:44,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:53:44,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:44,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:53:44,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:53:44,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:44,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34681,1681527209133': 2023-04-15 02:53:44,775 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34681,1681527209133' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-15 02:53:44,775 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-15 02:53:44,775 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-15 02:53:44,775 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-15 02:53:44,775 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-15 02:53:44,775 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-15 02:53:44,776 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c44270c3412c22e2c805f466aae6ff56, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:53:44,777 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:53:44,777 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34681] ipc.CallRunner(144): callId: 196 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:38530 deadline: 1681527284776, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c44270c3412c22e2c805f466aae6ff56, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:44,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:53:44,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:53:44,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:53:44,776 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:53:44,777 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-15 02:53:44,777 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:53:44,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:53:44,777 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-15 02:53:44,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-15 02:53:44,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:53:44,778 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:44,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:53:44,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:53:44,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:53:44,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:53:44,787 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:44,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:44,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:53:44,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:53:44,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:44,791 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-15 02:53:44,791 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:44,791 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-15 02:53:44,791 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-15 02:53:44,791 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-15 02:53:44,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-15 02:53:44,791 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-15 02:53:44,791 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-15 02:53:44,791 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:53:44,791 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-15 02:53:44,791 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:53:44,791 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-15 02:53:44,791 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-15 02:53:44,791 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:44,791 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:53:44,791 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:53:44,791 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:53:44,792 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-15 02:53:44,792 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-15 02:53:44,891 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-15 02:53:44,892 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-15 02:53:44,893 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-15 02:53:44,894 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-15 02:53:44,894 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-15 02:53:44,894 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-15 02:53:44,894 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-15 02:53:44,894 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-15 02:53:44,894 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-15 02:53:44,895 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:53:44,895 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:53:44,896 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-15 02:53:44,896 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:44,896 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-15 02:53:44,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-15 02:53:44,896 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:44,896 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-15 02:53:44,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:53:44,896 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:53:44,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-15 02:53:44,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:53:44,897 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-15 02:53:44,897 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-15 02:53:44,897 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-15 02:53:44,897 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-15 02:53:44,897 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-15 02:53:44,897 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-15 02:53:44,897 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-15 02:53:44,897 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681527213260.c44270c3412c22e2c805f466aae6ff56. 2023-04-15 02:53:44,897 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681527213260.c44270c3412c22e2c805f466aae6ff56. started... 2023-04-15 02:53:44,897 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681527213260.c44270c3412c22e2c805f466aae6ff56. as already flushing 2023-04-15 02:53:44,897 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681527213260.c44270c3412c22e2c805f466aae6ff56. 2023-04-15 02:53:44,897 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-15 02:53:44,898 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-15 02:53:44,898 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:44,898 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-15 02:53:44,898 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34681,1681527209133' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-15 02:53:44,898 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:53:44,898 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:44,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:44,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:53:44,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:53:44,899 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:53:44,899 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-15 02:53:44,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:53:44,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:53:44,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:53:44,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:44,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:53:44,900 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34681,1681527209133' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-15 02:53:44,900 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1ce5afcd[Count = 0] remaining members to acquire global barrier 2023-04-15 02:53:44,900 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-15 02:53:44,900 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:53:44,901 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:53:44,901 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:53:44,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:53:44,901 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-15 02:53:44,901 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-15 02:53:44,901 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34681,1681527209133' in zk 2023-04-15 02:53:44,901 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:44,901 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-15 02:53:44,902 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:44,902 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-15 02:53:44,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:44,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:53:44,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:53:44,902 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-15 02:53:44,902 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-15 02:53:44,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:53:44,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:53:44,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:53:44,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:44,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:53:44,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:53:44,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:44,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34681,1681527209133': 2023-04-15 02:53:44,904 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34681,1681527209133' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-15 02:53:44,904 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-15 02:53:44,904 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-15 02:53:44,904 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-15 02:53:44,904 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-15 02:53:44,904 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-15 02:53:44,906 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:53:44,906 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:53:44,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:53:44,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:53:44,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:53:44,906 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:53:44,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:53:44,906 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-15 02:53:44,906 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-15 02:53:44,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:53:44,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-15 02:53:44,906 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:44,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:53:44,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:53:44,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:53:44,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:53:44,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:44,912 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:44,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:53:44,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:53:44,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:44,923 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:44,923 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:53:44,923 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:53:44,924 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-15 02:53:44,924 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:44,924 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-15 02:53:44,924 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-15 02:53:44,924 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-15 02:53:44,924 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-15 02:53:44,924 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-15 02:53:44,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-15 02:53:44,924 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:53:44,924 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:53:44,924 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:53:44,924 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-15 02:53:44,924 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-15 02:53:44,924 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-15 02:53:44,924 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-15 02:53:44,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-15 02:53:45,024 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-15 02:53:45,025 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-15 02:53:45,026 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-15 02:53:45,027 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-15 02:53:45,027 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-15 02:53:45,027 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-15 02:53:45,027 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-15 02:53:45,027 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-15 02:53:45,027 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-15 02:53:45,028 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:53:45,028 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:53:45,029 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-15 02:53:45,029 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-15 02:53:45,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-15 02:53:45,029 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:45,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:53:45,029 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:45,029 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-15 02:53:45,029 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:53:45,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-15 02:53:45,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:53:45,029 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-15 02:53:45,030 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-15 02:53:45,030 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-15 02:53:45,030 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-15 02:53:45,030 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-15 02:53:45,030 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-15 02:53:45,030 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-15 02:53:45,030 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-15 02:53:45,030 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681527213260.c44270c3412c22e2c805f466aae6ff56. 2023-04-15 02:53:45,030 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681527213260.c44270c3412c22e2c805f466aae6ff56. started... 2023-04-15 02:53:45,030 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681527213260.c44270c3412c22e2c805f466aae6ff56. as already flushing 2023-04-15 02:53:45,030 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681527213260.c44270c3412c22e2c805f466aae6ff56. 2023-04-15 02:53:45,030 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-15 02:53:45,030 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-15 02:53:45,030 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:45,030 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-15 02:53:45,030 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34681,1681527209133' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-15 02:53:45,031 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:45,031 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:53:45,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:45,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:53:45,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:53:45,032 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:53:45,032 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-15 02:53:45,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:53:45,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:53:45,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:53:45,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:45,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:53:45,033 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34681,1681527209133' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-15 02:53:45,033 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@d02e99a[Count = 0] remaining members to acquire global barrier 2023-04-15 02:53:45,033 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-15 02:53:45,033 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:53:45,034 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:45,034 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-15 02:53:45,034 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:53:45,034 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:53:45,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:53:45,034 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-15 02:53:45,034 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-15 02:53:45,034 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34681,1681527209133' in zk 2023-04-15 02:53:45,035 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:45,035 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-15 02:53:45,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:45,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:53:45,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:53:45,035 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-15 02:53:45,035 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-15 02:53:45,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:53:45,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:53:45,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:53:45,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:45,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:53:45,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:53:45,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:45,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34681,1681527209133': 2023-04-15 02:53:45,037 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34681,1681527209133' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-15 02:53:45,037 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-15 02:53:45,037 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-15 02:53:45,037 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-15 02:53:45,037 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-15 02:53:45,037 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-15 02:53:45,038 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:53:45,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:53:45,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:53:45,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:53:45,038 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:53:45,038 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-15 02:53:45,038 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:53:45,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:53:45,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:53:45,038 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:45,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:53:45,039 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-15 02:53:45,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-15 02:53:45,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:53:45,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:53:45,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:53:45,039 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:45,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:45,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:53:45,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:53:45,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:45,041 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:45,041 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:53:45,041 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:53:45,041 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:45,041 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-15 02:53:45,041 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-15 02:53:45,041 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-15 02:53:45,041 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:53:45,041 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-15 02:53:45,041 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:53:45,041 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-15 02:53:45,041 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-15 02:53:45,041 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-15 02:53:45,041 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-15 02:53:45,041 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:53:45,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-15 02:53:45,041 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-15 02:53:45,042 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-15 02:53:45,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-15 02:53:45,081 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c44270c3412c22e2c805f466aae6ff56, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:53:45,081 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34681] ipc.CallRunner(144): callId: 198 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:38530 deadline: 1681527285079, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c44270c3412c22e2c805f466aae6ff56, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:45,142 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-15 02:53:45,142 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-15 02:53:45,144 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-15 02:53:45,145 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-15 02:53:45,145 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-15 02:53:45,145 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-15 02:53:45,145 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-15 02:53:45,145 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-15 02:53:45,145 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-15 02:53:45,145 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:53:45,145 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:53:45,146 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-15 02:53:45,146 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:45,146 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-15 02:53:45,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-15 02:53:45,146 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:45,146 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-15 02:53:45,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:53:45,147 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:53:45,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-15 02:53:45,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:53:45,147 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-15 02:53:45,147 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-15 02:53:45,147 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-15 02:53:45,147 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-15 02:53:45,147 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-15 02:53:45,147 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-15 02:53:45,147 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-15 02:53:45,147 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-15 02:53:45,147 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681527213260.c44270c3412c22e2c805f466aae6ff56. 2023-04-15 02:53:45,147 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681527213260.c44270c3412c22e2c805f466aae6ff56. started... 2023-04-15 02:53:45,147 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681527213260.c44270c3412c22e2c805f466aae6ff56. as already flushing 2023-04-15 02:53:45,147 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681527213260.c44270c3412c22e2c805f466aae6ff56. 2023-04-15 02:53:45,148 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-15 02:53:45,148 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-15 02:53:45,148 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:45,148 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-15 02:53:45,148 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34681,1681527209133' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-15 02:53:45,148 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:53:45,148 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:45,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:45,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:53:45,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:53:45,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:53:45,150 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=302 (bloomFilter=true), to=hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/c44270c3412c22e2c805f466aae6ff56/.tmp/B/613efbbcf3014af29a33f60fe1db0ef2 2023-04-15 02:53:45,150 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:53:45,150 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-15 02:53:45,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:53:45,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:53:45,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:45,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:53:45,151 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34681,1681527209133' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-15 02:53:45,151 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@27030a0e[Count = 0] remaining members to acquire global barrier 2023-04-15 02:53:45,151 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-15 02:53:45,151 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:53:45,153 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:45,153 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-15 02:53:45,158 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:53:45,158 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:53:45,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:53:45,158 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-15 02:53:45,158 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-15 02:53:45,158 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34681,1681527209133' in zk 2023-04-15 02:53:45,163 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=302 (bloomFilter=true), to=hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/c44270c3412c22e2c805f466aae6ff56/.tmp/C/d189626d0b484141936ae0d230907be8 2023-04-15 02:53:45,165 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:45,165 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-15 02:53:45,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:45,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:53:45,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:53:45,165 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-15 02:53:45,165 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-15 02:53:45,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:53:45,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:53:45,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:53:45,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:45,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:53:45,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:53:45,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:45,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34681,1681527209133': 2023-04-15 02:53:45,167 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34681,1681527209133' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-15 02:53:45,167 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-15 02:53:45,168 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-15 02:53:45,168 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-15 02:53:45,168 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-15 02:53:45,168 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-15 02:53:45,168 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/c44270c3412c22e2c805f466aae6ff56/.tmp/A/9b3a32d29fc74086b895f2308fd5a598 as hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/c44270c3412c22e2c805f466aae6ff56/A/9b3a32d29fc74086b895f2308fd5a598 2023-04-15 02:53:45,169 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:53:45,169 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:53:45,169 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-15 02:53:45,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:53:45,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:53:45,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:53:45,169 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:53:45,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:53:45,169 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:45,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:53:45,169 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-15 02:53:45,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-15 02:53:45,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:53:45,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:53:45,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:53:45,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:53:45,171 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:45,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:45,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:53:45,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:53:45,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:45,175 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/c44270c3412c22e2c805f466aae6ff56/A/9b3a32d29fc74086b895f2308fd5a598, entries=200, sequenceid=302, filesize=14.3 K 2023-04-15 02:53:45,177 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/c44270c3412c22e2c805f466aae6ff56/.tmp/B/613efbbcf3014af29a33f60fe1db0ef2 as hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/c44270c3412c22e2c805f466aae6ff56/B/613efbbcf3014af29a33f60fe1db0ef2 2023-04-15 02:53:45,177 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-15 02:53:45,177 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-15 02:53:45,177 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-15 02:53:45,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-15 02:53:45,177 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-15 02:53:45,177 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-15 02:53:45,177 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:45,177 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-15 02:53:45,178 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:53:45,178 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:53:45,178 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:45,178 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:53:45,178 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:53:45,178 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:53:45,178 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-15 02:53:45,178 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-15 02:53:45,178 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-15 02:53:45,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-15 02:53:45,178 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-15 02:53:45,184 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/c44270c3412c22e2c805f466aae6ff56/B/613efbbcf3014af29a33f60fe1db0ef2, entries=150, sequenceid=302, filesize=11.9 K 2023-04-15 02:53:45,185 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/c44270c3412c22e2c805f466aae6ff56/.tmp/C/d189626d0b484141936ae0d230907be8 as hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/c44270c3412c22e2c805f466aae6ff56/C/d189626d0b484141936ae0d230907be8 2023-04-15 02:53:45,191 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/c44270c3412c22e2c805f466aae6ff56/C/d189626d0b484141936ae0d230907be8, entries=150, sequenceid=302, filesize=11.9 K 2023-04-15 02:53:45,193 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=134.18 KB/137400 for c44270c3412c22e2c805f466aae6ff56 in 866ms, sequenceid=302, compaction requested=true 2023-04-15 02:53:45,193 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for c44270c3412c22e2c805f466aae6ff56: 2023-04-15 02:53:45,193 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-15 02:53:45,193 DEBUG [RS:0;jenkins-hbase20:34681-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 6 store files, 0 compacting, 6 eligible, 16 blocking 2023-04-15 02:53:45,193 DEBUG [RS:0;jenkins-hbase20:34681-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 6 store files, 0 compacting, 6 eligible, 16 blocking 2023-04-15 02:53:45,193 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-15 02:53:45,194 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-15 02:53:45,197 DEBUG [RS:0;jenkins-hbase20:34681-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 6 files of size 73122 starting at candidate #0 after considering 10 permutations with 10 in ratio 2023-04-15 02:53:45,197 DEBUG [RS:0;jenkins-hbase20:34681-shortCompactions-0] regionserver.HStore(1912): c44270c3412c22e2c805f466aae6ff56/A is initiating minor compaction (all files) 2023-04-15 02:53:45,197 INFO [RS:0;jenkins-hbase20:34681-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of c44270c3412c22e2c805f466aae6ff56/A in TestAcidGuarantees,,1681527213260.c44270c3412c22e2c805f466aae6ff56. 2023-04-15 02:53:45,197 INFO [RS:0;jenkins-hbase20:34681-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/c44270c3412c22e2c805f466aae6ff56/A/8f7b03202b8040ad85cab6a1a5522b63, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/c44270c3412c22e2c805f466aae6ff56/A/3fecd097e4dc418788d8e03e486d995f, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/c44270c3412c22e2c805f466aae6ff56/A/90066fdb8faf41cd90972274f2085d6c, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/c44270c3412c22e2c805f466aae6ff56/A/ae40346c39e94bcdb9eb81fbc0a744f2, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/c44270c3412c22e2c805f466aae6ff56/A/c0f7d1d451b94af2b158f512a2c089ed, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/c44270c3412c22e2c805f466aae6ff56/A/9b3a32d29fc74086b895f2308fd5a598] into tmpdir=hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/c44270c3412c22e2c805f466aae6ff56/.tmp, totalSize=71.4 K 2023-04-15 02:53:45,198 DEBUG [RS:0;jenkins-hbase20:34681-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 6 files of size 70682 starting at candidate #0 after considering 10 permutations with 10 in ratio 2023-04-15 02:53:45,198 DEBUG [RS:0;jenkins-hbase20:34681-longCompactions-0] regionserver.HStore(1912): c44270c3412c22e2c805f466aae6ff56/B is initiating minor compaction (all files) 2023-04-15 02:53:45,198 DEBUG [RS:0;jenkins-hbase20:34681-shortCompactions-0] compactions.Compactor(207): Compacting 8f7b03202b8040ad85cab6a1a5522b63, keycount=150, bloomtype=ROW, size=12.2 K, encoding=NONE, compression=NONE, seqNum=197, earliestPutTs=1681527221180 2023-04-15 02:53:45,198 INFO [RS:0;jenkins-hbase20:34681-longCompactions-0] regionserver.HRegion(2259): Starting compaction of c44270c3412c22e2c805f466aae6ff56/B in TestAcidGuarantees,,1681527213260.c44270c3412c22e2c805f466aae6ff56. 2023-04-15 02:53:45,198 INFO [RS:0;jenkins-hbase20:34681-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/c44270c3412c22e2c805f466aae6ff56/B/033565ea1a184d06b070bb8636f464a2, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/c44270c3412c22e2c805f466aae6ff56/B/ee3572e477764367b4360e7da2e9b6b9, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/c44270c3412c22e2c805f466aae6ff56/B/a517dd6eae5d4726a3b1947f94b7de28, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/c44270c3412c22e2c805f466aae6ff56/B/6e8954bf57a44d1598c74e296874d020, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/c44270c3412c22e2c805f466aae6ff56/B/575038b7953e422695ac1f08a68ff389, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/c44270c3412c22e2c805f466aae6ff56/B/613efbbcf3014af29a33f60fe1db0ef2] into tmpdir=hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/c44270c3412c22e2c805f466aae6ff56/.tmp, totalSize=69.0 K 2023-04-15 02:53:45,199 DEBUG [RS:0;jenkins-hbase20:34681-shortCompactions-0] compactions.Compactor(207): Compacting 3fecd097e4dc418788d8e03e486d995f, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=234, earliestPutTs=1681527221221 2023-04-15 02:53:45,199 DEBUG [RS:0;jenkins-hbase20:34681-longCompactions-0] compactions.Compactor(207): Compacting 033565ea1a184d06b070bb8636f464a2, keycount=150, bloomtype=ROW, size=12.2 K, encoding=NONE, compression=NONE, seqNum=197, earliestPutTs=1681527221180 2023-04-15 02:53:45,199 DEBUG [RS:0;jenkins-hbase20:34681-longCompactions-0] compactions.Compactor(207): Compacting ee3572e477764367b4360e7da2e9b6b9, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=234, earliestPutTs=1681527221221 2023-04-15 02:53:45,199 DEBUG [RS:0;jenkins-hbase20:34681-shortCompactions-0] compactions.Compactor(207): Compacting 90066fdb8faf41cd90972274f2085d6c, keycount=100, bloomtype=ROW, size=9.4 K, encoding=NONE, compression=NONE, seqNum=245, earliestPutTs=1681527222337 2023-04-15 02:53:45,199 DEBUG [RS:0;jenkins-hbase20:34681-longCompactions-0] compactions.Compactor(207): Compacting a517dd6eae5d4726a3b1947f94b7de28, keycount=100, bloomtype=ROW, size=9.4 K, encoding=NONE, compression=NONE, seqNum=245, earliestPutTs=1681527222337 2023-04-15 02:53:45,200 DEBUG [RS:0;jenkins-hbase20:34681-shortCompactions-0] compactions.Compactor(207): Compacting ae40346c39e94bcdb9eb81fbc0a744f2, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=282, earliestPutTs=1681527223047 2023-04-15 02:53:45,200 DEBUG [RS:0;jenkins-hbase20:34681-longCompactions-0] compactions.Compactor(207): Compacting 6e8954bf57a44d1598c74e296874d020, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=282, earliestPutTs=1681527223047 2023-04-15 02:53:45,201 DEBUG [RS:0;jenkins-hbase20:34681-shortCompactions-0] compactions.Compactor(207): Compacting c0f7d1d451b94af2b158f512a2c089ed, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=292, earliestPutTs=1681527223683 2023-04-15 02:53:45,201 DEBUG [RS:0;jenkins-hbase20:34681-longCompactions-0] compactions.Compactor(207): Compacting 575038b7953e422695ac1f08a68ff389, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=292, earliestPutTs=1681527223683 2023-04-15 02:53:45,201 DEBUG [RS:0;jenkins-hbase20:34681-shortCompactions-0] compactions.Compactor(207): Compacting 9b3a32d29fc74086b895f2308fd5a598, keycount=200, bloomtype=ROW, size=14.3 K, encoding=NONE, compression=NONE, seqNum=302, earliestPutTs=1681527224314 2023-04-15 02:53:45,201 DEBUG [RS:0;jenkins-hbase20:34681-longCompactions-0] compactions.Compactor(207): Compacting 613efbbcf3014af29a33f60fe1db0ef2, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=302, earliestPutTs=1681527224314 2023-04-15 02:53:45,221 INFO [RS:0;jenkins-hbase20:34681-shortCompactions-0] throttle.PressureAwareThroughputController(145): c44270c3412c22e2c805f466aae6ff56#A#compaction#54 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-15 02:53:45,222 INFO [RS:0;jenkins-hbase20:34681-longCompactions-0] throttle.PressureAwareThroughputController(145): c44270c3412c22e2c805f466aae6ff56#B#compaction#55 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-15 02:53:45,249 DEBUG [RS:0;jenkins-hbase20:34681-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/c44270c3412c22e2c805f466aae6ff56/.tmp/A/6f6754556026426b882980c6173c34d5 as hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/c44270c3412c22e2c805f466aae6ff56/A/6f6754556026426b882980c6173c34d5 2023-04-15 02:53:45,255 INFO [RS:0;jenkins-hbase20:34681-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 6 (all) file(s) in c44270c3412c22e2c805f466aae6ff56/A of c44270c3412c22e2c805f466aae6ff56 into 6f6754556026426b882980c6173c34d5(size=12.5 K), total size for store is 12.5 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-15 02:53:45,255 DEBUG [RS:0;jenkins-hbase20:34681-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for c44270c3412c22e2c805f466aae6ff56: 2023-04-15 02:53:45,255 INFO [RS:0;jenkins-hbase20:34681-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681527213260.c44270c3412c22e2c805f466aae6ff56., storeName=c44270c3412c22e2c805f466aae6ff56/A, priority=10, startTime=1681527225193; duration=0sec 2023-04-15 02:53:45,255 DEBUG [RS:0;jenkins-hbase20:34681-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-15 02:53:45,255 DEBUG [RS:0;jenkins-hbase20:34681-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 6 store files, 0 compacting, 6 eligible, 16 blocking 2023-04-15 02:53:45,258 DEBUG [RS:0;jenkins-hbase20:34681-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 6 files of size 70682 starting at candidate #0 after considering 10 permutations with 10 in ratio 2023-04-15 02:53:45,258 DEBUG [RS:0;jenkins-hbase20:34681-shortCompactions-0] regionserver.HStore(1912): c44270c3412c22e2c805f466aae6ff56/C is initiating minor compaction (all files) 2023-04-15 02:53:45,258 INFO [RS:0;jenkins-hbase20:34681-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of c44270c3412c22e2c805f466aae6ff56/C in TestAcidGuarantees,,1681527213260.c44270c3412c22e2c805f466aae6ff56. 2023-04-15 02:53:45,259 INFO [RS:0;jenkins-hbase20:34681-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/c44270c3412c22e2c805f466aae6ff56/C/4cb2f6c7cea1426cb832a4e7203a86bc, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/c44270c3412c22e2c805f466aae6ff56/C/ff9445b047e14cd4ace2d569eb140ab6, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/c44270c3412c22e2c805f466aae6ff56/C/935006ea70d34e00b33b6556dfd065fd, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/c44270c3412c22e2c805f466aae6ff56/C/c4954ab54de54450a33fc1030524f6ed, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/c44270c3412c22e2c805f466aae6ff56/C/d10899e00fe64a74810f1c8fe7034cc5, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/c44270c3412c22e2c805f466aae6ff56/C/d189626d0b484141936ae0d230907be8] into tmpdir=hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/c44270c3412c22e2c805f466aae6ff56/.tmp, totalSize=69.0 K 2023-04-15 02:53:45,259 DEBUG [RS:0;jenkins-hbase20:34681-shortCompactions-0] compactions.Compactor(207): Compacting 4cb2f6c7cea1426cb832a4e7203a86bc, keycount=150, bloomtype=ROW, size=12.2 K, encoding=NONE, compression=NONE, seqNum=197, earliestPutTs=1681527221180 2023-04-15 02:53:45,262 DEBUG [RS:0;jenkins-hbase20:34681-shortCompactions-0] compactions.Compactor(207): Compacting ff9445b047e14cd4ace2d569eb140ab6, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=234, earliestPutTs=1681527221221 2023-04-15 02:53:45,263 DEBUG [RS:0;jenkins-hbase20:34681-shortCompactions-0] compactions.Compactor(207): Compacting 935006ea70d34e00b33b6556dfd065fd, keycount=100, bloomtype=ROW, size=9.4 K, encoding=NONE, compression=NONE, seqNum=245, earliestPutTs=1681527222337 2023-04-15 02:53:45,263 DEBUG [RS:0;jenkins-hbase20:34681-shortCompactions-0] compactions.Compactor(207): Compacting c4954ab54de54450a33fc1030524f6ed, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=282, earliestPutTs=1681527223047 2023-04-15 02:53:45,264 DEBUG [RS:0;jenkins-hbase20:34681-shortCompactions-0] compactions.Compactor(207): Compacting d10899e00fe64a74810f1c8fe7034cc5, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=292, earliestPutTs=1681527223683 2023-04-15 02:53:45,264 DEBUG [RS:0;jenkins-hbase20:34681-shortCompactions-0] compactions.Compactor(207): Compacting d189626d0b484141936ae0d230907be8, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=302, earliestPutTs=1681527224314 2023-04-15 02:53:45,278 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-15 02:53:45,279 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-15 02:53:45,279 INFO [RS:0;jenkins-hbase20:34681-shortCompactions-0] throttle.PressureAwareThroughputController(145): c44270c3412c22e2c805f466aae6ff56#C#compaction#56 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-15 02:53:45,281 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-15 02:53:45,286 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-15 02:53:45,286 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-15 02:53:45,286 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-15 02:53:45,286 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-15 02:53:45,286 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-15 02:53:45,286 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-15 02:53:45,287 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:53:45,287 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:53:45,287 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:45,287 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-15 02:53:45,288 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-15 02:53:45,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-15 02:53:45,288 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:45,288 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-15 02:53:45,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:53:45,289 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:53:45,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-15 02:53:45,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:53:45,289 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-15 02:53:45,297 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-15 02:53:45,298 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-15 02:53:45,298 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-15 02:53:45,298 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-15 02:53:45,298 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-15 02:53:45,298 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-15 02:53:45,298 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-15 02:53:45,298 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681527213260.c44270c3412c22e2c805f466aae6ff56. 2023-04-15 02:53:45,298 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681527213260.c44270c3412c22e2c805f466aae6ff56. started... 2023-04-15 02:53:45,298 INFO [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing c44270c3412c22e2c805f466aae6ff56 3/3 column families, dataSize=134.18 KB heapSize=384.98 KB 2023-04-15 02:53:45,299 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c44270c3412c22e2c805f466aae6ff56, store=A 2023-04-15 02:53:45,299 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-15 02:53:45,299 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c44270c3412c22e2c805f466aae6ff56, store=B 2023-04-15 02:53:45,299 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-15 02:53:45,299 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c44270c3412c22e2c805f466aae6ff56, store=C 2023-04-15 02:53:45,299 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-15 02:53:45,308 INFO [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=340 (bloomFilter=true), to=hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/c44270c3412c22e2c805f466aae6ff56/.tmp/A/107ad8c17894439e9194631ffc9b3c4d 2023-04-15 02:53:45,374 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34681] regionserver.HRegion(9158): Flush requested on c44270c3412c22e2c805f466aae6ff56 2023-04-15 02:53:45,374 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681527213260.c44270c3412c22e2c805f466aae6ff56. as already flushing 2023-04-15 02:53:45,393 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c44270c3412c22e2c805f466aae6ff56, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:53:45,393 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34681] ipc.CallRunner(144): callId: 95 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:38504 deadline: 1681527285388, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c44270c3412c22e2c805f466aae6ff56, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:45,394 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c44270c3412c22e2c805f466aae6ff56, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:53:45,394 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34681] ipc.CallRunner(144): callId: 95 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:38542 deadline: 1681527285388, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c44270c3412c22e2c805f466aae6ff56, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:45,395 DEBUG [Thread-141] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4168 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c44270c3412c22e2c805f466aae6ff56, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_2' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1681527213260.c44270c3412c22e2c805f466aae6ff56., hostname=jenkins-hbase20.apache.org,34681,1681527209133, seqNum=2, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c44270c3412c22e2c805f466aae6ff56, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor31.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c44270c3412c22e2c805f466aae6ff56, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-15 02:53:45,399 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c44270c3412c22e2c805f466aae6ff56, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:53:45,399 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34681] ipc.CallRunner(144): callId: 101 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:38520 deadline: 1681527285393, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c44270c3412c22e2c805f466aae6ff56, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:45,497 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c44270c3412c22e2c805f466aae6ff56, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:53:45,498 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34681] ipc.CallRunner(144): callId: 97 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:38542 deadline: 1681527285496, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c44270c3412c22e2c805f466aae6ff56, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:45,501 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c44270c3412c22e2c805f466aae6ff56, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:53:45,501 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34681] ipc.CallRunner(144): callId: 103 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:38520 deadline: 1681527285500, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c44270c3412c22e2c805f466aae6ff56, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:45,561 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c44270c3412c22e2c805f466aae6ff56, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:53:45,561 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34681] ipc.CallRunner(144): callId: 49 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:38540 deadline: 1681527285558, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c44270c3412c22e2c805f466aae6ff56, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:45,563 DEBUG [Thread-143] client.RpcRetryingCallerImpl(129): Call exception, tries=7, retries=16, started=8202 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c44270c3412c22e2c805f466aae6ff56, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_2' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1681527213260.c44270c3412c22e2c805f466aae6ff56., hostname=jenkins-hbase20.apache.org,34681,1681527209133, seqNum=2, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c44270c3412c22e2c805f466aae6ff56, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor31.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c44270c3412c22e2c805f466aae6ff56, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-15 02:53:45,589 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c44270c3412c22e2c805f466aae6ff56, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:53:45,589 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34681] ipc.CallRunner(144): callId: 200 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:38530 deadline: 1681527285586, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c44270c3412c22e2c805f466aae6ff56, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:45,642 DEBUG [RS:0;jenkins-hbase20:34681-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/c44270c3412c22e2c805f466aae6ff56/.tmp/B/26642d181a934868b18a7d75c282341b as hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/c44270c3412c22e2c805f466aae6ff56/B/26642d181a934868b18a7d75c282341b 2023-04-15 02:53:45,648 INFO [RS:0;jenkins-hbase20:34681-longCompactions-0] regionserver.HStore(1652): Completed compaction of 6 (all) file(s) in c44270c3412c22e2c805f466aae6ff56/B of c44270c3412c22e2c805f466aae6ff56 into 26642d181a934868b18a7d75c282341b(size=12.5 K), total size for store is 12.5 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-15 02:53:45,648 DEBUG [RS:0;jenkins-hbase20:34681-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for c44270c3412c22e2c805f466aae6ff56: 2023-04-15 02:53:45,648 INFO [RS:0;jenkins-hbase20:34681-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681527213260.c44270c3412c22e2c805f466aae6ff56., storeName=c44270c3412c22e2c805f466aae6ff56/B, priority=10, startTime=1681527225193; duration=0sec 2023-04-15 02:53:45,648 DEBUG [RS:0;jenkins-hbase20:34681-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-15 02:53:45,701 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c44270c3412c22e2c805f466aae6ff56, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:53:45,701 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34681] ipc.CallRunner(144): callId: 99 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:38542 deadline: 1681527285700, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c44270c3412c22e2c805f466aae6ff56, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:45,703 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c44270c3412c22e2c805f466aae6ff56, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:53:45,704 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34681] ipc.CallRunner(144): callId: 105 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:38520 deadline: 1681527285703, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c44270c3412c22e2c805f466aae6ff56, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:45,704 DEBUG [RS:0;jenkins-hbase20:34681-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/c44270c3412c22e2c805f466aae6ff56/.tmp/C/bc739284553c434bbe2f22067bfb697a as hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/c44270c3412c22e2c805f466aae6ff56/C/bc739284553c434bbe2f22067bfb697a 2023-04-15 02:53:45,710 INFO [RS:0;jenkins-hbase20:34681-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 6 (all) file(s) in c44270c3412c22e2c805f466aae6ff56/C of c44270c3412c22e2c805f466aae6ff56 into bc739284553c434bbe2f22067bfb697a(size=12.5 K), total size for store is 12.5 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-15 02:53:45,710 DEBUG [RS:0;jenkins-hbase20:34681-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for c44270c3412c22e2c805f466aae6ff56: 2023-04-15 02:53:45,711 INFO [RS:0;jenkins-hbase20:34681-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681527213260.c44270c3412c22e2c805f466aae6ff56., storeName=c44270c3412c22e2c805f466aae6ff56/C, priority=10, startTime=1681527225194; duration=0sec 2023-04-15 02:53:45,711 DEBUG [RS:0;jenkins-hbase20:34681-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-15 02:53:45,725 INFO [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=340 (bloomFilter=true), to=hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/c44270c3412c22e2c805f466aae6ff56/.tmp/B/4c9e2f7065754ea4a844f9954a828880 2023-04-15 02:53:46,005 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c44270c3412c22e2c805f466aae6ff56, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:53:46,005 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34681] ipc.CallRunner(144): callId: 101 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:38542 deadline: 1681527286004, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c44270c3412c22e2c805f466aae6ff56, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:46,006 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c44270c3412c22e2c805f466aae6ff56, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:53:46,007 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34681] ipc.CallRunner(144): callId: 107 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:38520 deadline: 1681527286006, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c44270c3412c22e2c805f466aae6ff56, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:46,149 INFO [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=340 (bloomFilter=true), to=hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/c44270c3412c22e2c805f466aae6ff56/.tmp/C/c706a717a77e466ab0d6d968caccf6ce 2023-04-15 02:53:46,154 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/c44270c3412c22e2c805f466aae6ff56/.tmp/A/107ad8c17894439e9194631ffc9b3c4d as hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/c44270c3412c22e2c805f466aae6ff56/A/107ad8c17894439e9194631ffc9b3c4d 2023-04-15 02:53:46,160 INFO [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/c44270c3412c22e2c805f466aae6ff56/A/107ad8c17894439e9194631ffc9b3c4d, entries=150, sequenceid=340, filesize=11.9 K 2023-04-15 02:53:46,160 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/c44270c3412c22e2c805f466aae6ff56/.tmp/B/4c9e2f7065754ea4a844f9954a828880 as hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/c44270c3412c22e2c805f466aae6ff56/B/4c9e2f7065754ea4a844f9954a828880 2023-04-15 02:53:46,166 INFO [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/c44270c3412c22e2c805f466aae6ff56/B/4c9e2f7065754ea4a844f9954a828880, entries=150, sequenceid=340, filesize=11.9 K 2023-04-15 02:53:46,166 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/c44270c3412c22e2c805f466aae6ff56/.tmp/C/c706a717a77e466ab0d6d968caccf6ce as hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/c44270c3412c22e2c805f466aae6ff56/C/c706a717a77e466ab0d6d968caccf6ce 2023-04-15 02:53:46,176 INFO [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/c44270c3412c22e2c805f466aae6ff56/C/c706a717a77e466ab0d6d968caccf6ce, entries=150, sequenceid=340, filesize=11.9 K 2023-04-15 02:53:46,177 INFO [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~134.18 KB/137400, heapSize ~384.94 KB/394176, currentSize=46.96 KB/48090 for c44270c3412c22e2c805f466aae6ff56 in 879ms, sequenceid=340, compaction requested=false 2023-04-15 02:53:46,178 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for c44270c3412c22e2c805f466aae6ff56: 2023-04-15 02:53:46,178 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681527213260.c44270c3412c22e2c805f466aae6ff56. 2023-04-15 02:53:46,178 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-15 02:53:46,178 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-15 02:53:46,178 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:46,178 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-15 02:53:46,178 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34681,1681527209133' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-15 02:53:46,180 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:46,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:46,180 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:53:46,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:53:46,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:53:46,180 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:53:46,180 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-15 02:53:46,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:53:46,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:53:46,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:53:46,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:46,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:53:46,181 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34681,1681527209133' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-15 02:53:46,181 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1c19849f[Count = 0] remaining members to acquire global barrier 2023-04-15 02:53:46,181 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-15 02:53:46,182 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:53:46,182 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:53:46,182 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:53:46,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:53:46,182 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-15 02:53:46,182 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-15 02:53:46,182 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34681,1681527209133' in zk 2023-04-15 02:53:46,183 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:46,183 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-15 02:53:46,184 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-15 02:53:46,184 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-15 02:53:46,184 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-15 02:53:46,184 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:46,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:46,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:53:46,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:53:46,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:53:46,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:53:46,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:53:46,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:46,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:53:46,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:53:46,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:46,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34681,1681527209133': 2023-04-15 02:53:46,187 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34681,1681527209133' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-15 02:53:46,187 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-15 02:53:46,187 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-15 02:53:46,187 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-15 02:53:46,187 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-15 02:53:46,187 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-15 02:53:46,188 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:53:46,188 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:53:46,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:53:46,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:53:46,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:53:46,188 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:53:46,188 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-15 02:53:46,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:53:46,189 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-15 02:53:46,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-15 02:53:46,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:53:46,190 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:46,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:53:46,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:53:46,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:53:46,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:53:46,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:46,191 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:46,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:53:46,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:53:46,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:46,194 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-15 02:53:46,194 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-15 02:53:46,194 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-15 02:53:46,194 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:46,194 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:53:46,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-15 02:53:46,194 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-15 02:53:46,194 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:53:46,194 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-15 02:53:46,194 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-15 02:53:46,194 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:46,194 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:53:46,194 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:53:46,194 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:53:46,194 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-15 02:53:46,195 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-15 02:53:46,195 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-15 02:53:46,195 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-15 02:53:46,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-15 02:53:46,295 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-15 02:53:46,295 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-15 02:53:46,297 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-15 02:53:46,298 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-15 02:53:46,298 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-15 02:53:46,298 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-15 02:53:46,298 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-15 02:53:46,298 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-15 02:53:46,298 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-15 02:53:46,298 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:53:46,299 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:53:46,299 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:46,299 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-15 02:53:46,299 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-15 02:53:46,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-15 02:53:46,300 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:46,300 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-15 02:53:46,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:53:46,300 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:53:46,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-15 02:53:46,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:53:46,301 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-15 02:53:46,306 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-15 02:53:46,306 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-15 02:53:46,306 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-15 02:53:46,306 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-15 02:53:46,306 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-15 02:53:46,310 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-15 02:53:46,310 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-15 02:53:46,310 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681527213260.c44270c3412c22e2c805f466aae6ff56. 2023-04-15 02:53:46,310 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681527213260.c44270c3412c22e2c805f466aae6ff56. started... 2023-04-15 02:53:46,310 INFO [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing c44270c3412c22e2c805f466aae6ff56 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-15 02:53:46,311 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c44270c3412c22e2c805f466aae6ff56, store=A 2023-04-15 02:53:46,311 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-15 02:53:46,311 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c44270c3412c22e2c805f466aae6ff56, store=B 2023-04-15 02:53:46,311 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-15 02:53:46,311 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c44270c3412c22e2c805f466aae6ff56, store=C 2023-04-15 02:53:46,311 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-15 02:53:46,511 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34681] regionserver.HRegion(9158): Flush requested on c44270c3412c22e2c805f466aae6ff56 2023-04-15 02:53:46,511 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681527213260.c44270c3412c22e2c805f466aae6ff56. as already flushing 2023-04-15 02:53:46,541 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681527226531, max timestamp=1681527226539Num uniques 859; 2023-04-15 02:53:46,541 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681527226531, max timestamp=1681527226539Num uniques 859; 2023-04-15 02:53:46,541 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-4] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-15 02:53:46,541 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-9] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-15 02:53:46,542 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681527226531, max timestamp=1681527226539Num uniques 859; 2023-04-15 02:53:46,543 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-3] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-15 02:53:46,577 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c44270c3412c22e2c805f466aae6ff56, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:53:46,577 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34681] ipc.CallRunner(144): callId: 125 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:38520 deadline: 1681527286574, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c44270c3412c22e2c805f466aae6ff56, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:46,577 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c44270c3412c22e2c805f466aae6ff56, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:53:46,578 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34681] ipc.CallRunner(144): callId: 121 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:38542 deadline: 1681527286575, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c44270c3412c22e2c805f466aae6ff56, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:46,597 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c44270c3412c22e2c805f466aae6ff56, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:53:46,597 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34681] ipc.CallRunner(144): callId: 202 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:38530 deadline: 1681527286597, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c44270c3412c22e2c805f466aae6ff56, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:46,679 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c44270c3412c22e2c805f466aae6ff56, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:53:46,680 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34681] ipc.CallRunner(144): callId: 127 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:38520 deadline: 1681527286679, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c44270c3412c22e2c805f466aae6ff56, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:46,680 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c44270c3412c22e2c805f466aae6ff56, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:53:46,680 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34681] ipc.CallRunner(144): callId: 123 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:38542 deadline: 1681527286679, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c44270c3412c22e2c805f466aae6ff56, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:46,733 INFO [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=352 (bloomFilter=true), to=hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/c44270c3412c22e2c805f466aae6ff56/.tmp/A/ff96f2210e8741199ea6a6a64e0596ad 2023-04-15 02:53:46,884 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c44270c3412c22e2c805f466aae6ff56, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:53:46,885 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34681] ipc.CallRunner(144): callId: 129 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:38520 deadline: 1681527286881, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c44270c3412c22e2c805f466aae6ff56, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:46,885 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c44270c3412c22e2c805f466aae6ff56, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:53:46,885 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34681] ipc.CallRunner(144): callId: 125 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:38542 deadline: 1681527286882, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c44270c3412c22e2c805f466aae6ff56, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:47,149 INFO [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=352 (bloomFilter=true), to=hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/c44270c3412c22e2c805f466aae6ff56/.tmp/B/3fadaf364e634d6f808b345ecf287298 2023-04-15 02:53:47,188 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c44270c3412c22e2c805f466aae6ff56, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:53:47,188 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34681] ipc.CallRunner(144): callId: 127 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:38542 deadline: 1681527287186, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c44270c3412c22e2c805f466aae6ff56, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:47,189 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c44270c3412c22e2c805f466aae6ff56, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:53:47,189 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34681] ipc.CallRunner(144): callId: 131 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:38520 deadline: 1681527287188, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c44270c3412c22e2c805f466aae6ff56, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:47,561 INFO [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=352 (bloomFilter=true), to=hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/c44270c3412c22e2c805f466aae6ff56/.tmp/C/42d7ae1915bf4283935d2781d54de6d7 2023-04-15 02:53:47,566 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/c44270c3412c22e2c805f466aae6ff56/.tmp/A/ff96f2210e8741199ea6a6a64e0596ad as hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/c44270c3412c22e2c805f466aae6ff56/A/ff96f2210e8741199ea6a6a64e0596ad 2023-04-15 02:53:47,571 INFO [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/c44270c3412c22e2c805f466aae6ff56/A/ff96f2210e8741199ea6a6a64e0596ad, entries=150, sequenceid=352, filesize=11.9 K 2023-04-15 02:53:47,572 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/c44270c3412c22e2c805f466aae6ff56/.tmp/B/3fadaf364e634d6f808b345ecf287298 as hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/c44270c3412c22e2c805f466aae6ff56/B/3fadaf364e634d6f808b345ecf287298 2023-04-15 02:53:47,577 INFO [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/c44270c3412c22e2c805f466aae6ff56/B/3fadaf364e634d6f808b345ecf287298, entries=150, sequenceid=352, filesize=11.9 K 2023-04-15 02:53:47,578 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/c44270c3412c22e2c805f466aae6ff56/.tmp/C/42d7ae1915bf4283935d2781d54de6d7 as hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/c44270c3412c22e2c805f466aae6ff56/C/42d7ae1915bf4283935d2781d54de6d7 2023-04-15 02:53:47,583 INFO [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/c44270c3412c22e2c805f466aae6ff56/C/42d7ae1915bf4283935d2781d54de6d7, entries=150, sequenceid=352, filesize=11.9 K 2023-04-15 02:53:47,583 INFO [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=134.18 KB/137400 for c44270c3412c22e2c805f466aae6ff56 in 1273ms, sequenceid=352, compaction requested=true 2023-04-15 02:53:47,584 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for c44270c3412c22e2c805f466aae6ff56: 2023-04-15 02:53:47,584 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681527213260.c44270c3412c22e2c805f466aae6ff56. 2023-04-15 02:53:47,584 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-15 02:53:47,584 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-15 02:53:47,584 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:47,584 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-15 02:53:47,584 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34681,1681527209133' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-15 02:53:47,585 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:53:47,585 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:47,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:47,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:53:47,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:53:47,585 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:53:47,585 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-15 02:53:47,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:53:47,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:53:47,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:53:47,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:47,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:53:47,586 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34681,1681527209133' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-15 02:53:47,586 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2db3cb16[Count = 0] remaining members to acquire global barrier 2023-04-15 02:53:47,586 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-15 02:53:47,586 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:53:47,587 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:53:47,587 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:53:47,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:53:47,587 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-15 02:53:47,587 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:47,587 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-15 02:53:47,587 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-15 02:53:47,587 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34681,1681527209133' in zk 2023-04-15 02:53:47,588 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:47,588 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-15 02:53:47,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:47,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:53:47,588 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-15 02:53:47,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:53:47,588 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-15 02:53:47,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:53:47,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:53:47,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:53:47,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:47,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:53:47,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:53:47,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:47,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34681,1681527209133': 2023-04-15 02:53:47,590 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34681,1681527209133' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-15 02:53:47,590 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-15 02:53:47,590 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-15 02:53:47,590 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-15 02:53:47,590 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-15 02:53:47,590 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-15 02:53:47,591 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:53:47,591 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:53:47,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:53:47,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:53:47,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:53:47,591 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:53:47,591 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-15 02:53:47,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:53:47,592 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-15 02:53:47,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-15 02:53:47,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:53:47,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:53:47,592 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:47,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:53:47,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:53:47,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:53:47,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:47,593 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:47,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:53:47,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:53:47,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:47,595 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:47,595 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-15 02:53:47,595 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-15 02:53:47,595 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-15 02:53:47,595 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:53:47,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-15 02:53:47,595 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-15 02:53:47,595 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-15 02:53:47,595 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-15 02:53:47,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-15 02:53:47,595 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:53:47,595 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-15 02:53:47,596 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-15 02:53:47,596 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:47,596 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:53:47,596 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:53:47,596 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:53:47,596 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-15 02:53:47,596 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-15 02:53:47,693 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34681] regionserver.HRegion(9158): Flush requested on c44270c3412c22e2c805f466aae6ff56 2023-04-15 02:53:47,693 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing c44270c3412c22e2c805f466aae6ff56 3/3 column families, dataSize=140.89 KB heapSize=405.61 KB 2023-04-15 02:53:47,693 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c44270c3412c22e2c805f466aae6ff56, store=A 2023-04-15 02:53:47,693 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=3, new segment=null 2023-04-15 02:53:47,693 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c44270c3412c22e2c805f466aae6ff56, store=B 2023-04-15 02:53:47,693 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=3, new segment=null 2023-04-15 02:53:47,693 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c44270c3412c22e2c805f466aae6ff56, store=C 2023-04-15 02:53:47,693 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=3, new segment=null 2023-04-15 02:53:47,696 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-15 02:53:47,697 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-15 02:53:47,698 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-15 02:53:47,699 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-15 02:53:47,699 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-15 02:53:47,699 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-15 02:53:47,699 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-15 02:53:47,699 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-15 02:53:47,699 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-15 02:53:47,701 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:53:47,701 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:53:47,702 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:47,702 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-15 02:53:47,702 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-15 02:53:47,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-15 02:53:47,702 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:47,702 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-15 02:53:47,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:53:47,703 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:53:47,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-15 02:53:47,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:53:47,703 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-15 02:53:47,703 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-15 02:53:47,703 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-15 02:53:47,704 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-15 02:53:47,704 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-15 02:53:47,704 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-15 02:53:47,706 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-15 02:53:47,706 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681527213260.c44270c3412c22e2c805f466aae6ff56. 2023-04-15 02:53:47,706 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-15 02:53:47,706 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681527213260.c44270c3412c22e2c805f466aae6ff56. started... 2023-04-15 02:53:47,706 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681527213260.c44270c3412c22e2c805f466aae6ff56. as already flushing 2023-04-15 02:53:47,706 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681527213260.c44270c3412c22e2c805f466aae6ff56. 2023-04-15 02:53:47,706 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-15 02:53:47,706 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-15 02:53:47,706 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:47,706 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-15 02:53:47,706 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34681,1681527209133' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-15 02:53:47,707 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:53:47,707 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:47,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:47,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:53:47,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:53:47,708 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:53:47,708 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-15 02:53:47,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:53:47,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:53:47,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:53:47,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:47,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:53:47,709 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34681,1681527209133' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-15 02:53:47,709 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@37e7a903[Count = 0] remaining members to acquire global barrier 2023-04-15 02:53:47,709 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-15 02:53:47,709 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:53:47,709 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c44270c3412c22e2c805f466aae6ff56, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:53:47,709 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34681] ipc.CallRunner(144): callId: 132 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:38542 deadline: 1681527287706, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c44270c3412c22e2c805f466aae6ff56, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:47,710 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:53:47,710 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:53:47,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:53:47,710 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-15 02:53:47,710 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-15 02:53:47,710 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34681,1681527209133' in zk 2023-04-15 02:53:47,710 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:47,710 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-15 02:53:47,711 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:47,711 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-15 02:53:47,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:47,711 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-15 02:53:47,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:53:47,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:53:47,711 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-15 02:53:47,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:53:47,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:53:47,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:53:47,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:47,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:53:47,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:53:47,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:47,713 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c44270c3412c22e2c805f466aae6ff56, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:53:47,713 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34681] ipc.CallRunner(144): callId: 137 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:38520 deadline: 1681527287710, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c44270c3412c22e2c805f466aae6ff56, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:47,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34681,1681527209133': 2023-04-15 02:53:47,713 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34681,1681527209133' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-15 02:53:47,713 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-15 02:53:47,713 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-15 02:53:47,713 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-15 02:53:47,713 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-15 02:53:47,713 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-15 02:53:47,714 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:53:47,714 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:53:47,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:53:47,714 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:53:47,714 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-15 02:53:47,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:53:47,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:53:47,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:53:47,715 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-15 02:53:47,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:53:47,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-15 02:53:47,715 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:47,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:53:47,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:53:47,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:53:47,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:53:47,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:47,716 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:47,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:53:47,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:53:47,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:47,718 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-15 02:53:47,718 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-15 02:53:47,718 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-15 02:53:47,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-15 02:53:47,718 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:47,718 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:53:47,718 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:53:47,718 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:47,718 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:53:47,718 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:53:47,719 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:53:47,719 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-15 02:53:47,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-15 02:53:47,719 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-15 02:53:47,719 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-15 02:53:47,719 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-15 02:53:47,720 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-15 02:53:47,720 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-15 02:53:47,720 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-15 02:53:47,812 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c44270c3412c22e2c805f466aae6ff56, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:53:47,812 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34681] ipc.CallRunner(144): callId: 134 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:38542 deadline: 1681527287810, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c44270c3412c22e2c805f466aae6ff56, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:47,816 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c44270c3412c22e2c805f466aae6ff56, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:53:47,816 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34681] ipc.CallRunner(144): callId: 139 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:38520 deadline: 1681527287814, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c44270c3412c22e2c805f466aae6ff56, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:47,820 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-15 02:53:47,820 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-15 02:53:47,821 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-15 02:53:47,822 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-15 02:53:47,822 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-15 02:53:47,822 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-15 02:53:47,822 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-15 02:53:47,822 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-15 02:53:47,823 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-15 02:53:47,823 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:53:47,823 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:53:47,824 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:47,824 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-15 02:53:47,824 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-15 02:53:47,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-15 02:53:47,824 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:47,824 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-15 02:53:47,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:53:47,824 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:53:47,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-15 02:53:47,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:53:47,825 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-15 02:53:47,825 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-15 02:53:47,825 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-15 02:53:47,825 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-15 02:53:47,825 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-15 02:53:47,825 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-15 02:53:47,831 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-15 02:53:47,831 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681527213260.c44270c3412c22e2c805f466aae6ff56. 2023-04-15 02:53:47,831 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-15 02:53:47,831 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681527213260.c44270c3412c22e2c805f466aae6ff56. started... 2023-04-15 02:53:47,831 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681527213260.c44270c3412c22e2c805f466aae6ff56. as already flushing 2023-04-15 02:53:47,832 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681527213260.c44270c3412c22e2c805f466aae6ff56. 2023-04-15 02:53:47,832 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-15 02:53:47,832 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-15 02:53:47,832 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:47,832 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-15 02:53:47,832 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34681,1681527209133' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-15 02:53:47,840 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:47,840 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:53:47,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:47,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:53:47,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:53:47,840 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:53:47,840 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-15 02:53:47,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:53:47,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:53:47,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:53:47,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:47,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:53:47,842 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34681,1681527209133' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-15 02:53:47,842 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@80b968b[Count = 0] remaining members to acquire global barrier 2023-04-15 02:53:47,842 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-15 02:53:47,842 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:53:47,844 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:53:47,844 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:53:47,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:53:47,844 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-15 02:53:47,844 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-15 02:53:47,844 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:47,845 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-15 02:53:47,845 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34681,1681527209133' in zk 2023-04-15 02:53:47,845 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-15 02:53:47,845 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:47,845 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-15 02:53:47,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:47,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:53:47,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:53:47,845 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-15 02:53:47,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:53:47,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:53:47,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:53:47,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:47,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:53:47,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:53:47,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:47,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34681,1681527209133': 2023-04-15 02:53:47,848 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34681,1681527209133' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-15 02:53:47,848 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-15 02:53:47,848 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-15 02:53:47,848 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-15 02:53:47,848 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-15 02:53:47,848 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-15 02:53:47,849 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:53:47,849 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-15 02:53:47,849 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:53:47,849 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:53:47,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:53:47,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:53:47,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:53:47,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:53:47,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:53:47,850 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:47,850 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-15 02:53:47,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-15 02:53:47,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:53:47,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:53:47,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:53:47,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:53:47,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:47,851 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:47,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:53:47,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:53:47,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:47,852 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:47,852 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-15 02:53:47,853 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:53:47,853 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:53:47,853 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:47,853 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:53:47,853 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:53:47,853 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:53:47,853 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-15 02:53:47,853 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-15 02:53:47,853 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-15 02:53:47,853 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-15 02:53:47,853 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-15 02:53:47,853 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-15 02:53:47,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-15 02:53:47,853 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-15 02:53:47,853 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-15 02:53:47,853 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-15 02:53:47,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-15 02:53:47,953 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-15 02:53:47,954 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-15 02:53:47,955 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-15 02:53:47,956 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-15 02:53:47,956 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-15 02:53:47,957 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-15 02:53:47,957 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-15 02:53:47,957 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-15 02:53:47,957 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-15 02:53:47,957 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:53:47,957 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:53:47,958 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-15 02:53:47,958 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:47,958 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-15 02:53:47,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-15 02:53:47,958 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:47,958 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-15 02:53:47,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:53:47,959 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:53:47,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-15 02:53:47,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:53:47,959 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-15 02:53:47,959 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-15 02:53:47,959 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-15 02:53:47,959 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-15 02:53:47,959 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-15 02:53:47,960 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-15 02:53:47,961 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-15 02:53:47,961 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-15 02:53:47,961 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681527213260.c44270c3412c22e2c805f466aae6ff56. 2023-04-15 02:53:47,961 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681527213260.c44270c3412c22e2c805f466aae6ff56. started... 2023-04-15 02:53:47,961 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681527213260.c44270c3412c22e2c805f466aae6ff56. as already flushing 2023-04-15 02:53:47,961 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681527213260.c44270c3412c22e2c805f466aae6ff56. 2023-04-15 02:53:47,961 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-15 02:53:47,961 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-15 02:53:47,961 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:47,962 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-15 02:53:47,962 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34681,1681527209133' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-15 02:53:47,962 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:53:47,963 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:47,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:47,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:53:47,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:53:47,963 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:53:47,963 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-15 02:53:47,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:53:47,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:53:47,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:53:47,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:47,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:53:47,964 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34681,1681527209133' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-15 02:53:47,964 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-15 02:53:47,964 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5866d669[Count = 0] remaining members to acquire global barrier 2023-04-15 02:53:47,964 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:53:47,965 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:53:47,965 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:53:47,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:53:47,965 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-15 02:53:47,965 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-15 02:53:47,965 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34681,1681527209133' in zk 2023-04-15 02:53:47,965 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:47,965 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-15 02:53:47,966 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:47,966 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-15 02:53:47,966 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-15 02:53:47,966 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-15 02:53:47,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:47,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:53:47,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:53:47,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:53:47,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:53:47,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:53:47,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:47,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:53:47,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:53:47,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:47,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34681,1681527209133': 2023-04-15 02:53:47,969 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34681,1681527209133' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-15 02:53:47,969 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-15 02:53:47,969 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-15 02:53:47,969 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-15 02:53:47,969 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-15 02:53:47,969 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-15 02:53:47,970 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:53:47,970 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-15 02:53:47,970 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:53:47,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:53:47,970 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:53:47,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:53:47,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:53:47,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:53:47,971 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-15 02:53:47,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-15 02:53:47,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:53:47,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:53:47,972 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:47,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:53:47,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:53:47,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:53:47,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:47,973 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:47,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:53:47,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:53:47,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:47,982 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-15 02:53:47,982 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-15 02:53:47,982 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-15 02:53:47,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-15 02:53:47,982 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:47,982 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:53:47,982 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:53:47,982 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:47,982 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:53:47,982 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:53:47,982 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:53:47,983 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-15 02:53:47,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-15 02:53:47,983 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-15 02:53:47,983 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-15 02:53:47,983 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-15 02:53:47,983 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-15 02:53:47,983 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-15 02:53:48,015 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c44270c3412c22e2c805f466aae6ff56, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:53:48,015 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34681] ipc.CallRunner(144): callId: 136 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:38542 deadline: 1681527288014, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c44270c3412c22e2c805f466aae6ff56, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:48,023 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c44270c3412c22e2c805f466aae6ff56, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:53:48,023 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34681] ipc.CallRunner(144): callId: 141 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:38520 deadline: 1681527288018, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c44270c3412c22e2c805f466aae6ff56, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:48,084 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-15 02:53:48,084 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-15 02:53:48,085 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-15 02:53:48,086 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-15 02:53:48,086 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-15 02:53:48,087 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-15 02:53:48,087 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-15 02:53:48,087 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-15 02:53:48,087 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-15 02:53:48,087 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:53:48,087 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:53:48,088 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-15 02:53:48,088 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:48,088 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-15 02:53:48,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-15 02:53:48,088 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:48,088 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-15 02:53:48,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:53:48,088 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:53:48,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-15 02:53:48,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:53:48,089 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-15 02:53:48,089 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-15 02:53:48,089 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-15 02:53:48,089 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-15 02:53:48,089 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-15 02:53:48,089 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-15 02:53:48,089 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-15 02:53:48,089 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-15 02:53:48,089 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681527213260.c44270c3412c22e2c805f466aae6ff56. 2023-04-15 02:53:48,089 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681527213260.c44270c3412c22e2c805f466aae6ff56. started... 2023-04-15 02:53:48,089 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681527213260.c44270c3412c22e2c805f466aae6ff56. as already flushing 2023-04-15 02:53:48,089 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681527213260.c44270c3412c22e2c805f466aae6ff56. 2023-04-15 02:53:48,090 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-15 02:53:48,090 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-15 02:53:48,090 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:48,090 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-15 02:53:48,090 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34681,1681527209133' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-15 02:53:48,091 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:53:48,091 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:48,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:48,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:53:48,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:53:48,091 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:53:48,091 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-15 02:53:48,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:53:48,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:53:48,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:53:48,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:48,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:53:48,092 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34681,1681527209133' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-15 02:53:48,092 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6bbe4d08[Count = 0] remaining members to acquire global barrier 2023-04-15 02:53:48,092 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-15 02:53:48,092 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:53:48,093 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:53:48,093 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:53:48,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:53:48,093 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-15 02:53:48,093 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-15 02:53:48,093 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34681,1681527209133' in zk 2023-04-15 02:53:48,093 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:48,093 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-15 02:53:48,094 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:48,094 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-15 02:53:48,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:48,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:53:48,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:53:48,094 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-15 02:53:48,094 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-15 02:53:48,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:53:48,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:53:48,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:53:48,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:48,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:53:48,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:53:48,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:48,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34681,1681527209133': 2023-04-15 02:53:48,096 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34681,1681527209133' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-15 02:53:48,096 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-15 02:53:48,096 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-15 02:53:48,096 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-15 02:53:48,096 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-15 02:53:48,096 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-15 02:53:48,097 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:53:48,097 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:53:48,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:53:48,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:53:48,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:53:48,097 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:53:48,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:53:48,097 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-15 02:53:48,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:53:48,097 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:48,097 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-15 02:53:48,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-15 02:53:48,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:53:48,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:53:48,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:53:48,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:53:48,099 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:48,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:48,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:53:48,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:53:48,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:48,101 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:48,101 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:53:48,101 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:53:48,101 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:48,101 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-15 02:53:48,101 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-15 02:53:48,101 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-15 02:53:48,101 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-15 02:53:48,101 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-15 02:53:48,101 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:53:48,102 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:53:48,102 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-15 02:53:48,102 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-15 02:53:48,102 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-15 02:53:48,102 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:53:48,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-15 02:53:48,102 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-15 02:53:48,102 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-15 02:53:48,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-15 02:53:48,107 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=390 (bloomFilter=true), to=hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/c44270c3412c22e2c805f466aae6ff56/.tmp/A/c14d9273a0a44d31ac175cdf798bf17b 2023-04-15 02:53:48,202 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-15 02:53:48,202 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-15 02:53:48,204 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-15 02:53:48,205 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-15 02:53:48,205 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-15 02:53:48,205 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-15 02:53:48,205 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-15 02:53:48,205 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-15 02:53:48,205 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-15 02:53:48,205 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:53:48,205 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:53:48,207 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-15 02:53:48,207 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:48,207 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-15 02:53:48,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-15 02:53:48,208 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:48,208 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-15 02:53:48,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:53:48,208 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:53:48,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-15 02:53:48,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:53:48,208 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-15 02:53:48,208 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-15 02:53:48,208 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-15 02:53:48,209 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-15 02:53:48,209 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-15 02:53:48,209 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-15 02:53:48,209 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-15 02:53:48,209 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-15 02:53:48,209 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681527213260.c44270c3412c22e2c805f466aae6ff56. 2023-04-15 02:53:48,209 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681527213260.c44270c3412c22e2c805f466aae6ff56. started... 2023-04-15 02:53:48,209 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681527213260.c44270c3412c22e2c805f466aae6ff56. as already flushing 2023-04-15 02:53:48,209 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681527213260.c44270c3412c22e2c805f466aae6ff56. 2023-04-15 02:53:48,209 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-15 02:53:48,209 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-15 02:53:48,209 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:48,209 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-15 02:53:48,209 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34681,1681527209133' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-15 02:53:48,210 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:48,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:48,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:53:48,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:53:48,210 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:53:48,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:53:48,211 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:53:48,211 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-15 02:53:48,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:53:48,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:53:48,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:48,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:53:48,212 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34681,1681527209133' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-15 02:53:48,212 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@619b5ecd[Count = 0] remaining members to acquire global barrier 2023-04-15 02:53:48,212 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-15 02:53:48,212 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:53:48,212 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:53:48,212 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:53:48,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:53:48,212 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-15 02:53:48,212 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:48,212 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-15 02:53:48,212 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-15 02:53:48,212 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34681,1681527209133' in zk 2023-04-15 02:53:48,214 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:48,214 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-15 02:53:48,214 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-15 02:53:48,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:48,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:53:48,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:53:48,214 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-15 02:53:48,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:53:48,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:53:48,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:53:48,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:48,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:53:48,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:53:48,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:48,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34681,1681527209133': 2023-04-15 02:53:48,217 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34681,1681527209133' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-15 02:53:48,217 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-15 02:53:48,218 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-15 02:53:48,218 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-15 02:53:48,218 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-15 02:53:48,218 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-15 02:53:48,219 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:53:48,219 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:53:48,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:53:48,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:53:48,219 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:53:48,219 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-15 02:53:48,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:53:48,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:53:48,219 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-15 02:53:48,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-15 02:53:48,219 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:48,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:53:48,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:53:48,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:53:48,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:53:48,220 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:48,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:53:48,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:48,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:53:48,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:53:48,222 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:48,222 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:53:48,222 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-15 02:53:48,222 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:53:48,222 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:48,222 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:53:48,222 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:53:48,222 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:53:48,222 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-15 02:53:48,222 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-15 02:53:48,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-15 02:53:48,222 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-15 02:53:48,222 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-15 02:53:48,222 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-15 02:53:48,222 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-15 02:53:48,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-15 02:53:48,222 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-15 02:53:48,222 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-15 02:53:48,223 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-15 02:53:48,318 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c44270c3412c22e2c805f466aae6ff56, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:53:48,318 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34681] ipc.CallRunner(144): callId: 138 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:38542 deadline: 1681527288318, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c44270c3412c22e2c805f466aae6ff56, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:48,323 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-15 02:53:48,323 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-15 02:53:48,324 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-15 02:53:48,325 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-15 02:53:48,325 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-15 02:53:48,325 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-15 02:53:48,325 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-15 02:53:48,325 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-15 02:53:48,325 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-15 02:53:48,326 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:53:48,326 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:53:48,327 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-15 02:53:48,327 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:48,327 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-15 02:53:48,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-15 02:53:48,327 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c44270c3412c22e2c805f466aae6ff56, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:53:48,327 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:48,327 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-15 02:53:48,327 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34681] ipc.CallRunner(144): callId: 143 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:38520 deadline: 1681527288325, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c44270c3412c22e2c805f466aae6ff56, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:48,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:53:48,327 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:53:48,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-15 02:53:48,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:53:48,328 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-15 02:53:48,328 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-15 02:53:48,328 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-15 02:53:48,328 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-15 02:53:48,328 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-15 02:53:48,328 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-15 02:53:48,328 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-15 02:53:48,328 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-15 02:53:48,328 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681527213260.c44270c3412c22e2c805f466aae6ff56. 2023-04-15 02:53:48,328 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681527213260.c44270c3412c22e2c805f466aae6ff56. started... 2023-04-15 02:53:48,328 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681527213260.c44270c3412c22e2c805f466aae6ff56. as already flushing 2023-04-15 02:53:48,328 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681527213260.c44270c3412c22e2c805f466aae6ff56. 2023-04-15 02:53:48,328 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-15 02:53:48,328 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-15 02:53:48,328 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:48,329 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-15 02:53:48,329 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34681,1681527209133' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-15 02:53:48,329 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:48,329 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:53:48,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:48,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:53:48,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:53:48,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:53:48,330 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:53:48,330 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-15 02:53:48,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:53:48,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:53:48,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:48,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:53:48,335 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34681,1681527209133' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-15 02:53:48,335 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@357033bb[Count = 0] remaining members to acquire global barrier 2023-04-15 02:53:48,335 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-15 02:53:48,335 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:53:48,336 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:53:48,336 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:53:48,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:53:48,336 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-15 02:53:48,336 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:48,336 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-15 02:53:48,336 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-15 02:53:48,336 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34681,1681527209133' in zk 2023-04-15 02:53:48,337 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:48,337 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-15 02:53:48,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:48,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:53:48,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:53:48,337 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-15 02:53:48,337 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-15 02:53:48,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:53:48,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:53:48,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:53:48,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:48,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:53:48,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:53:48,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:48,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34681,1681527209133': 2023-04-15 02:53:48,340 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34681,1681527209133' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-15 02:53:48,340 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-15 02:53:48,340 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-15 02:53:48,340 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-15 02:53:48,340 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-15 02:53:48,340 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-15 02:53:48,341 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:53:48,341 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-15 02:53:48,341 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:53:48,341 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:53:48,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:53:48,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:53:48,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:53:48,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:53:48,341 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-15 02:53:48,341 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:48,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:53:48,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-15 02:53:48,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:53:48,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:53:48,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:53:48,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:53:48,343 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:48,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:48,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:53:48,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:53:48,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:48,345 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:48,345 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-15 02:53:48,345 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:53:48,345 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-15 02:53:48,345 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-15 02:53:48,345 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-15 02:53:48,345 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-15 02:53:48,345 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-15 02:53:48,345 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-15 02:53:48,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-15 02:53:48,345 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:53:48,346 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:48,346 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-15 02:53:48,346 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:53:48,346 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-15 02:53:48,346 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-15 02:53:48,346 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:53:48,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-15 02:53:48,346 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:53:48,446 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-15 02:53:48,446 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-15 02:53:48,448 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-15 02:53:48,448 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-15 02:53:48,449 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-15 02:53:48,449 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-15 02:53:48,449 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-15 02:53:48,449 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-15 02:53:48,449 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-15 02:53:48,449 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:53:48,449 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:53:48,450 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-15 02:53:48,450 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:48,450 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-15 02:53:48,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-15 02:53:48,450 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:48,450 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-15 02:53:48,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:53:48,450 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:53:48,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-15 02:53:48,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:53:48,451 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-15 02:53:48,451 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-15 02:53:48,451 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-15 02:53:48,451 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-15 02:53:48,451 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-15 02:53:48,451 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-15 02:53:48,451 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-15 02:53:48,451 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-15 02:53:48,451 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681527213260.c44270c3412c22e2c805f466aae6ff56. 2023-04-15 02:53:48,451 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681527213260.c44270c3412c22e2c805f466aae6ff56. started... 2023-04-15 02:53:48,451 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681527213260.c44270c3412c22e2c805f466aae6ff56. as already flushing 2023-04-15 02:53:48,451 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681527213260.c44270c3412c22e2c805f466aae6ff56. 2023-04-15 02:53:48,451 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-15 02:53:48,451 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-15 02:53:48,452 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:48,452 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-15 02:53:48,452 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34681,1681527209133' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-15 02:53:48,452 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:48,452 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:53:48,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:48,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:53:48,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:53:48,453 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:53:48,453 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-15 02:53:48,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:53:48,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:53:48,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:53:48,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:48,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:53:48,454 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34681,1681527209133' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-15 02:53:48,454 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5bcb6ee2[Count = 0] remaining members to acquire global barrier 2023-04-15 02:53:48,454 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-15 02:53:48,454 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:53:48,455 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:53:48,455 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:53:48,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:53:48,455 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-15 02:53:48,455 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-15 02:53:48,455 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34681,1681527209133' in zk 2023-04-15 02:53:48,455 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:48,455 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-15 02:53:48,455 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-15 02:53:48,455 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-15 02:53:48,455 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:48,455 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-15 02:53:48,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:48,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:53:48,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:53:48,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:53:48,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:53:48,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:53:48,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:48,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:53:48,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:53:48,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:48,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34681,1681527209133': 2023-04-15 02:53:48,458 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34681,1681527209133' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-15 02:53:48,458 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-15 02:53:48,458 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-15 02:53:48,458 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-15 02:53:48,458 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-15 02:53:48,458 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-15 02:53:48,459 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:53:48,459 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:53:48,459 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:53:48,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:53:48,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:53:48,459 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-15 02:53:48,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:53:48,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:53:48,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:53:48,459 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:48,459 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-15 02:53:48,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-15 02:53:48,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:53:48,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:53:48,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:53:48,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:53:48,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:48,460 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:48,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:53:48,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:53:48,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:48,462 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:48,462 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-15 02:53:48,462 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:53:48,462 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-15 02:53:48,462 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:53:48,462 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-15 02:53:48,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-15 02:53:48,462 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:48,462 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:53:48,462 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-15 02:53:48,462 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-15 02:53:48,462 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:53:48,463 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:53:48,462 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-15 02:53:48,463 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-15 02:53:48,463 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-15 02:53:48,463 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-15 02:53:48,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-15 02:53:48,519 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=390 (bloomFilter=true), to=hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/c44270c3412c22e2c805f466aae6ff56/.tmp/B/7276d3ce7f4c4e78be690dabe67ac89c 2023-04-15 02:53:48,563 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-15 02:53:48,564 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-15 02:53:48,565 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-15 02:53:48,566 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-15 02:53:48,566 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-15 02:53:48,566 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-15 02:53:48,566 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-15 02:53:48,566 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-15 02:53:48,566 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-15 02:53:48,566 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:53:48,566 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:53:48,567 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-15 02:53:48,567 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:48,567 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-15 02:53:48,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-15 02:53:48,567 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:48,567 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-15 02:53:48,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:53:48,567 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:53:48,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-15 02:53:48,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:53:48,568 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-15 02:53:48,568 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-15 02:53:48,568 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-15 02:53:48,568 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-15 02:53:48,568 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-15 02:53:48,568 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-15 02:53:48,568 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-15 02:53:48,568 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-15 02:53:48,568 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681527213260.c44270c3412c22e2c805f466aae6ff56. 2023-04-15 02:53:48,568 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681527213260.c44270c3412c22e2c805f466aae6ff56. started... 2023-04-15 02:53:48,569 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681527213260.c44270c3412c22e2c805f466aae6ff56. as already flushing 2023-04-15 02:53:48,569 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681527213260.c44270c3412c22e2c805f466aae6ff56. 2023-04-15 02:53:48,569 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-15 02:53:48,569 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-15 02:53:48,569 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:48,569 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-15 02:53:48,569 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34681,1681527209133' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-15 02:53:48,569 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:48,569 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:53:48,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:48,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:53:48,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:53:48,570 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:53:48,570 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-15 02:53:48,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:53:48,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:53:48,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:53:48,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:48,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:53:48,571 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34681,1681527209133' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-15 02:53:48,571 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-15 02:53:48,571 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4284ae9e[Count = 0] remaining members to acquire global barrier 2023-04-15 02:53:48,571 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:53:48,572 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:53:48,572 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:53:48,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:53:48,572 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-15 02:53:48,572 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:48,572 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-15 02:53:48,572 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-15 02:53:48,572 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34681,1681527209133' in zk 2023-04-15 02:53:48,573 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-15 02:53:48,573 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:48,573 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-15 02:53:48,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:48,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:53:48,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:53:48,573 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-15 02:53:48,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:53:48,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:53:48,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:53:48,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:48,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:53:48,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:53:48,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:48,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34681,1681527209133': 2023-04-15 02:53:48,575 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34681,1681527209133' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-15 02:53:48,575 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-15 02:53:48,575 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-15 02:53:48,575 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-15 02:53:48,575 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-15 02:53:48,575 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-15 02:53:48,576 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:53:48,576 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:53:48,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:53:48,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:53:48,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:53:48,576 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:53:48,576 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-15 02:53:48,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:53:48,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:53:48,576 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-15 02:53:48,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-15 02:53:48,578 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:48,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:53:48,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:53:48,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:53:48,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:53:48,579 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:48,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:48,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:53:48,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:53:48,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:48,581 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:48,581 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:53:48,581 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:53:48,581 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-15 02:53:48,581 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:48,581 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-15 02:53:48,581 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-15 02:53:48,581 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-15 02:53:48,582 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-15 02:53:48,582 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-15 02:53:48,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-15 02:53:48,581 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:53:48,582 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:53:48,582 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:53:48,582 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-15 02:53:48,582 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-15 02:53:48,582 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-15 02:53:48,583 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-15 02:53:48,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-15 02:53:48,615 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c44270c3412c22e2c805f466aae6ff56, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:53:48,615 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34681] ipc.CallRunner(144): callId: 204 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:38530 deadline: 1681527288613, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c44270c3412c22e2c805f466aae6ff56, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:48,617 DEBUG [Thread-147] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4145 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c44270c3412c22e2c805f466aae6ff56, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_1' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1681527213260.c44270c3412c22e2c805f466aae6ff56., hostname=jenkins-hbase20.apache.org,34681,1681527209133, seqNum=2, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c44270c3412c22e2c805f466aae6ff56, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor31.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c44270c3412c22e2c805f466aae6ff56, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-15 02:53:48,682 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-15 02:53:48,683 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-15 02:53:48,684 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-15 02:53:48,685 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-15 02:53:48,685 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-15 02:53:48,685 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-15 02:53:48,685 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-15 02:53:48,685 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-15 02:53:48,685 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-15 02:53:48,686 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:53:48,686 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:53:48,686 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:48,686 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-15 02:53:48,686 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-15 02:53:48,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-15 02:53:48,687 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:48,687 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-15 02:53:48,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:53:48,687 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:53:48,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-15 02:53:48,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:53:48,687 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-15 02:53:48,687 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-15 02:53:48,687 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-15 02:53:48,688 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-15 02:53:48,688 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-15 02:53:48,688 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-15 02:53:48,688 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-15 02:53:48,688 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-15 02:53:48,688 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681527213260.c44270c3412c22e2c805f466aae6ff56. 2023-04-15 02:53:48,688 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681527213260.c44270c3412c22e2c805f466aae6ff56. started... 2023-04-15 02:53:48,688 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681527213260.c44270c3412c22e2c805f466aae6ff56. as already flushing 2023-04-15 02:53:48,688 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681527213260.c44270c3412c22e2c805f466aae6ff56. 2023-04-15 02:53:48,688 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-15 02:53:48,688 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-15 02:53:48,688 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:48,688 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-15 02:53:48,688 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34681,1681527209133' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-15 02:53:48,689 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:48,689 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:53:48,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:48,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:53:48,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:53:48,689 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:53:48,689 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-15 02:53:48,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:53:48,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:53:48,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:53:48,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:48,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:53:48,690 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34681,1681527209133' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-15 02:53:48,690 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@79d2fd35[Count = 0] remaining members to acquire global barrier 2023-04-15 02:53:48,690 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-15 02:53:48,690 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:53:48,691 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:53:48,691 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:53:48,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:53:48,691 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-15 02:53:48,691 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:48,691 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-15 02:53:48,691 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-15 02:53:48,691 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34681,1681527209133' in zk 2023-04-15 02:53:48,692 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:48,692 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-15 02:53:48,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:48,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:53:48,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:53:48,692 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-15 02:53:48,692 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-15 02:53:48,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:53:48,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:53:48,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:53:48,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:48,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:53:48,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:53:48,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:48,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34681,1681527209133': 2023-04-15 02:53:48,694 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34681,1681527209133' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-15 02:53:48,694 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-15 02:53:48,694 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-15 02:53:48,694 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-15 02:53:48,694 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-15 02:53:48,694 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-15 02:53:48,694 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:53:48,694 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:53:48,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:53:48,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:53:48,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:53:48,695 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:53:48,695 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-15 02:53:48,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:53:48,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:53:48,695 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:48,695 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-15 02:53:48,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-15 02:53:48,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:53:48,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:53:48,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:53:48,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:53:48,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:48,696 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:48,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:53:48,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:53:48,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:48,697 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:48,697 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-15 02:53:48,697 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:53:48,697 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-15 02:53:48,697 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-15 02:53:48,697 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-15 02:53:48,697 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-15 02:53:48,697 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-15 02:53:48,697 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:53:48,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-15 02:53:48,697 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-15 02:53:48,697 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:48,698 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:53:48,698 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:53:48,698 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:53:48,697 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-15 02:53:48,698 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-15 02:53:48,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-15 02:53:48,798 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-15 02:53:48,798 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-15 02:53:48,799 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-15 02:53:48,800 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-15 02:53:48,800 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-15 02:53:48,800 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-15 02:53:48,800 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-15 02:53:48,800 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-15 02:53:48,800 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-15 02:53:48,801 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:53:48,801 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:53:48,801 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-15 02:53:48,801 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:48,801 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-15 02:53:48,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-15 02:53:48,802 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:48,802 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-15 02:53:48,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:53:48,802 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:53:48,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-15 02:53:48,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:53:48,802 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-15 02:53:48,802 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-15 02:53:48,802 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-15 02:53:48,803 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-15 02:53:48,803 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-15 02:53:48,803 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-15 02:53:48,807 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-15 02:53:48,808 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-15 02:53:48,808 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681527213260.c44270c3412c22e2c805f466aae6ff56. 2023-04-15 02:53:48,808 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681527213260.c44270c3412c22e2c805f466aae6ff56. started... 2023-04-15 02:53:48,808 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681527213260.c44270c3412c22e2c805f466aae6ff56. as already flushing 2023-04-15 02:53:48,808 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681527213260.c44270c3412c22e2c805f466aae6ff56. 2023-04-15 02:53:48,808 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-15 02:53:48,808 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-15 02:53:48,808 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:48,808 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-15 02:53:48,808 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34681,1681527209133' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-15 02:53:48,810 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:48,810 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:53:48,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:48,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:53:48,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:53:48,811 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:53:48,811 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-15 02:53:48,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:53:48,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:53:48,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:53:48,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:48,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:53:48,812 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34681,1681527209133' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-15 02:53:48,812 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4d0bf45f[Count = 0] remaining members to acquire global barrier 2023-04-15 02:53:48,812 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-15 02:53:48,812 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:53:48,813 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:53:48,813 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:53:48,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:53:48,813 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:48,813 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-15 02:53:48,813 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-15 02:53:48,813 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-15 02:53:48,813 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34681,1681527209133' in zk 2023-04-15 02:53:48,814 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:48,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:48,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:53:48,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:53:48,814 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-15 02:53:48,814 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-15 02:53:48,814 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-15 02:53:48,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:53:48,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:53:48,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:53:48,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:48,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:53:48,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:53:48,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:48,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34681,1681527209133': 2023-04-15 02:53:48,816 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34681,1681527209133' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-15 02:53:48,816 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-15 02:53:48,816 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-15 02:53:48,816 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-15 02:53:48,816 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-15 02:53:48,817 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-15 02:53:48,817 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:53:48,817 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:53:48,817 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-15 02:53:48,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:53:48,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:53:48,817 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:53:48,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:53:48,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:53:48,818 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:48,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:53:48,818 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-15 02:53:48,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-15 02:53:48,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:53:48,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:53:48,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:53:48,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:53:48,819 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:48,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:48,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:53:48,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:53:48,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:48,820 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:48,820 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:53:48,820 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:53:48,820 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-15 02:53:48,820 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-15 02:53:48,820 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-15 02:53:48,820 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-15 02:53:48,820 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:48,820 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:53:48,820 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:53:48,820 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:53:48,820 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-15 02:53:48,820 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-15 02:53:48,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-15 02:53:48,820 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-15 02:53:48,821 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-15 02:53:48,821 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-15 02:53:48,821 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-15 02:53:48,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-15 02:53:48,824 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c44270c3412c22e2c805f466aae6ff56, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:53:48,824 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34681] ipc.CallRunner(144): callId: 140 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:38542 deadline: 1681527288820, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c44270c3412c22e2c805f466aae6ff56, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:48,832 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c44270c3412c22e2c805f466aae6ff56, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:53:48,832 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34681] ipc.CallRunner(144): callId: 145 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:38520 deadline: 1681527288829, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c44270c3412c22e2c805f466aae6ff56, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:48,921 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-15 02:53:48,921 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-15 02:53:48,923 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-15 02:53:48,924 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-15 02:53:48,924 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-15 02:53:48,924 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-15 02:53:48,924 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-15 02:53:48,924 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-15 02:53:48,924 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-15 02:53:48,924 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:53:48,924 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:53:48,925 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-15 02:53:48,925 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:48,925 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-15 02:53:48,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-15 02:53:48,925 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:48,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:53:48,925 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-15 02:53:48,926 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:53:48,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-15 02:53:48,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:53:48,926 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-15 02:53:48,926 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-15 02:53:48,926 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-15 02:53:48,926 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-15 02:53:48,926 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-15 02:53:48,926 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-15 02:53:48,927 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-15 02:53:48,927 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-15 02:53:48,927 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681527213260.c44270c3412c22e2c805f466aae6ff56. 2023-04-15 02:53:48,927 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681527213260.c44270c3412c22e2c805f466aae6ff56. started... 2023-04-15 02:53:48,927 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681527213260.c44270c3412c22e2c805f466aae6ff56. as already flushing 2023-04-15 02:53:48,927 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681527213260.c44270c3412c22e2c805f466aae6ff56. 2023-04-15 02:53:48,927 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-15 02:53:48,927 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-15 02:53:48,927 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:48,927 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-15 02:53:48,927 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34681,1681527209133' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-15 02:53:48,928 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:48,928 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:53:48,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:48,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:53:48,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:53:48,928 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:53:48,928 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-15 02:53:48,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:53:48,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:53:48,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:53:48,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:48,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:53:48,929 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34681,1681527209133' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-15 02:53:48,929 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-15 02:53:48,929 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4d373f62[Count = 0] remaining members to acquire global barrier 2023-04-15 02:53:48,929 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:53:48,930 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:53:48,930 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:53:48,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:53:48,930 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-15 02:53:48,930 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-15 02:53:48,930 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34681,1681527209133' in zk 2023-04-15 02:53:48,930 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:48,930 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-15 02:53:48,930 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:48,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:48,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:53:48,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:53:48,931 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-15 02:53:48,931 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-15 02:53:48,931 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-15 02:53:48,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:53:48,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:53:48,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:53:48,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:48,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:53:48,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:53:48,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:48,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34681,1681527209133': 2023-04-15 02:53:48,932 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34681,1681527209133' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-15 02:53:48,933 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=390 (bloomFilter=true), to=hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/c44270c3412c22e2c805f466aae6ff56/.tmp/C/fc51d87e600940bda48e00d50d6afe40 2023-04-15 02:53:48,933 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-15 02:53:48,933 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-15 02:53:48,933 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-15 02:53:48,933 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-15 02:53:48,933 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-15 02:53:48,934 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:53:48,934 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-15 02:53:48,934 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:53:48,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:53:48,935 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-15 02:53:48,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-15 02:53:48,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:53:48,938 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:53:48,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:53:48,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:53:48,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:53:48,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:53:48,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:53:48,941 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:48,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:53:48,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:53:48,942 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/c44270c3412c22e2c805f466aae6ff56/.tmp/A/c14d9273a0a44d31ac175cdf798bf17b as hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/c44270c3412c22e2c805f466aae6ff56/A/c14d9273a0a44d31ac175cdf798bf17b 2023-04-15 02:53:48,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:48,946 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:48,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:53:48,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:53:48,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:48,948 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/c44270c3412c22e2c805f466aae6ff56/A/c14d9273a0a44d31ac175cdf798bf17b, entries=200, sequenceid=390, filesize=14.3 K 2023-04-15 02:53:48,949 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/c44270c3412c22e2c805f466aae6ff56/.tmp/B/7276d3ce7f4c4e78be690dabe67ac89c as hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/c44270c3412c22e2c805f466aae6ff56/B/7276d3ce7f4c4e78be690dabe67ac89c 2023-04-15 02:53:48,949 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:48,949 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-15 02:53:48,949 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:53:48,949 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-15 02:53:48,949 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-15 02:53:48,949 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-15 02:53:48,949 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:53:48,949 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-15 02:53:48,949 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-15 02:53:48,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-15 02:53:48,949 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:48,950 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-15 02:53:48,950 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:53:48,950 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-15 02:53:48,950 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:53:48,950 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-15 02:53:48,950 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:53:48,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-15 02:53:48,960 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/c44270c3412c22e2c805f466aae6ff56/B/7276d3ce7f4c4e78be690dabe67ac89c, entries=150, sequenceid=390, filesize=11.9 K 2023-04-15 02:53:48,961 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/c44270c3412c22e2c805f466aae6ff56/.tmp/C/fc51d87e600940bda48e00d50d6afe40 as hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/c44270c3412c22e2c805f466aae6ff56/C/fc51d87e600940bda48e00d50d6afe40 2023-04-15 02:53:48,966 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/c44270c3412c22e2c805f466aae6ff56/C/fc51d87e600940bda48e00d50d6afe40, entries=150, sequenceid=390, filesize=11.9 K 2023-04-15 02:53:48,967 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~140.89 KB/144270, heapSize ~405.56 KB/415296, currentSize=40.25 KB/41220 for c44270c3412c22e2c805f466aae6ff56 in 1274ms, sequenceid=390, compaction requested=true 2023-04-15 02:53:48,967 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for c44270c3412c22e2c805f466aae6ff56: 2023-04-15 02:53:48,967 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-15 02:53:48,967 DEBUG [RS:0;jenkins-hbase20:34681-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-15 02:53:48,967 DEBUG [RS:0;jenkins-hbase20:34681-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-15 02:53:48,967 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-15 02:53:48,967 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-15 02:53:48,969 DEBUG [RS:0;jenkins-hbase20:34681-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 51798 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-15 02:53:48,969 DEBUG [RS:0;jenkins-hbase20:34681-shortCompactions-0] regionserver.HStore(1912): c44270c3412c22e2c805f466aae6ff56/A is initiating minor compaction (all files) 2023-04-15 02:53:48,969 DEBUG [RS:0;jenkins-hbase20:34681-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 49358 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-15 02:53:48,969 INFO [RS:0;jenkins-hbase20:34681-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of c44270c3412c22e2c805f466aae6ff56/A in TestAcidGuarantees,,1681527213260.c44270c3412c22e2c805f466aae6ff56. 2023-04-15 02:53:48,969 DEBUG [RS:0;jenkins-hbase20:34681-longCompactions-0] regionserver.HStore(1912): c44270c3412c22e2c805f466aae6ff56/B is initiating minor compaction (all files) 2023-04-15 02:53:48,969 INFO [RS:0;jenkins-hbase20:34681-longCompactions-0] regionserver.HRegion(2259): Starting compaction of c44270c3412c22e2c805f466aae6ff56/B in TestAcidGuarantees,,1681527213260.c44270c3412c22e2c805f466aae6ff56. 2023-04-15 02:53:48,969 INFO [RS:0;jenkins-hbase20:34681-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/c44270c3412c22e2c805f466aae6ff56/B/26642d181a934868b18a7d75c282341b, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/c44270c3412c22e2c805f466aae6ff56/B/4c9e2f7065754ea4a844f9954a828880, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/c44270c3412c22e2c805f466aae6ff56/B/3fadaf364e634d6f808b345ecf287298, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/c44270c3412c22e2c805f466aae6ff56/B/7276d3ce7f4c4e78be690dabe67ac89c] into tmpdir=hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/c44270c3412c22e2c805f466aae6ff56/.tmp, totalSize=48.2 K 2023-04-15 02:53:48,969 INFO [RS:0;jenkins-hbase20:34681-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/c44270c3412c22e2c805f466aae6ff56/A/6f6754556026426b882980c6173c34d5, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/c44270c3412c22e2c805f466aae6ff56/A/107ad8c17894439e9194631ffc9b3c4d, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/c44270c3412c22e2c805f466aae6ff56/A/ff96f2210e8741199ea6a6a64e0596ad, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/c44270c3412c22e2c805f466aae6ff56/A/c14d9273a0a44d31ac175cdf798bf17b] into tmpdir=hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/c44270c3412c22e2c805f466aae6ff56/.tmp, totalSize=50.6 K 2023-04-15 02:53:48,969 DEBUG [RS:0;jenkins-hbase20:34681-longCompactions-0] compactions.Compactor(207): Compacting 26642d181a934868b18a7d75c282341b, keycount=150, bloomtype=ROW, size=12.5 K, encoding=NONE, compression=NONE, seqNum=302, earliestPutTs=1681527224314 2023-04-15 02:53:48,969 DEBUG [RS:0;jenkins-hbase20:34681-shortCompactions-0] compactions.Compactor(207): Compacting 6f6754556026426b882980c6173c34d5, keycount=150, bloomtype=ROW, size=12.5 K, encoding=NONE, compression=NONE, seqNum=302, earliestPutTs=1681527224314 2023-04-15 02:53:48,970 DEBUG [RS:0;jenkins-hbase20:34681-longCompactions-0] compactions.Compactor(207): Compacting 4c9e2f7065754ea4a844f9954a828880, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=340, earliestPutTs=1681527224456 2023-04-15 02:53:48,970 DEBUG [RS:0;jenkins-hbase20:34681-shortCompactions-0] compactions.Compactor(207): Compacting 107ad8c17894439e9194631ffc9b3c4d, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=340, earliestPutTs=1681527224456 2023-04-15 02:53:48,970 DEBUG [RS:0;jenkins-hbase20:34681-longCompactions-0] compactions.Compactor(207): Compacting 3fadaf364e634d6f808b345ecf287298, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=352, earliestPutTs=1681527225383 2023-04-15 02:53:48,970 DEBUG [RS:0;jenkins-hbase20:34681-shortCompactions-0] compactions.Compactor(207): Compacting ff96f2210e8741199ea6a6a64e0596ad, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=352, earliestPutTs=1681527225383 2023-04-15 02:53:48,971 DEBUG [RS:0;jenkins-hbase20:34681-longCompactions-0] compactions.Compactor(207): Compacting 7276d3ce7f4c4e78be690dabe67ac89c, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=390, earliestPutTs=1681527226572 2023-04-15 02:53:48,971 DEBUG [RS:0;jenkins-hbase20:34681-shortCompactions-0] compactions.Compactor(207): Compacting c14d9273a0a44d31ac175cdf798bf17b, keycount=200, bloomtype=ROW, size=14.3 K, encoding=NONE, compression=NONE, seqNum=390, earliestPutTs=1681527226569 2023-04-15 02:53:48,985 INFO [RS:0;jenkins-hbase20:34681-longCompactions-0] throttle.PressureAwareThroughputController(145): c44270c3412c22e2c805f466aae6ff56#B#compaction#66 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-15 02:53:48,988 INFO [RS:0;jenkins-hbase20:34681-shortCompactions-0] throttle.PressureAwareThroughputController(145): c44270c3412c22e2c805f466aae6ff56#A#compaction#67 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-15 02:53:49,026 DEBUG [RS:0;jenkins-hbase20:34681-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/c44270c3412c22e2c805f466aae6ff56/.tmp/A/2573076332df46de97607613d26daddd as hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/c44270c3412c22e2c805f466aae6ff56/A/2573076332df46de97607613d26daddd 2023-04-15 02:53:49,034 INFO [RS:0;jenkins-hbase20:34681-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in c44270c3412c22e2c805f466aae6ff56/A of c44270c3412c22e2c805f466aae6ff56 into 2573076332df46de97607613d26daddd(size=12.6 K), total size for store is 12.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-15 02:53:49,035 DEBUG [RS:0;jenkins-hbase20:34681-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for c44270c3412c22e2c805f466aae6ff56: 2023-04-15 02:53:49,035 INFO [RS:0;jenkins-hbase20:34681-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681527213260.c44270c3412c22e2c805f466aae6ff56., storeName=c44270c3412c22e2c805f466aae6ff56/A, priority=12, startTime=1681527228967; duration=0sec 2023-04-15 02:53:49,035 DEBUG [RS:0;jenkins-hbase20:34681-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-15 02:53:49,035 DEBUG [RS:0;jenkins-hbase20:34681-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-15 02:53:49,037 DEBUG [RS:0;jenkins-hbase20:34681-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 49358 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-15 02:53:49,037 DEBUG [RS:0;jenkins-hbase20:34681-shortCompactions-0] regionserver.HStore(1912): c44270c3412c22e2c805f466aae6ff56/C is initiating minor compaction (all files) 2023-04-15 02:53:49,037 INFO [RS:0;jenkins-hbase20:34681-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of c44270c3412c22e2c805f466aae6ff56/C in TestAcidGuarantees,,1681527213260.c44270c3412c22e2c805f466aae6ff56. 2023-04-15 02:53:49,037 INFO [RS:0;jenkins-hbase20:34681-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/c44270c3412c22e2c805f466aae6ff56/C/bc739284553c434bbe2f22067bfb697a, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/c44270c3412c22e2c805f466aae6ff56/C/c706a717a77e466ab0d6d968caccf6ce, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/c44270c3412c22e2c805f466aae6ff56/C/42d7ae1915bf4283935d2781d54de6d7, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/c44270c3412c22e2c805f466aae6ff56/C/fc51d87e600940bda48e00d50d6afe40] into tmpdir=hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/c44270c3412c22e2c805f466aae6ff56/.tmp, totalSize=48.2 K 2023-04-15 02:53:49,038 DEBUG [RS:0;jenkins-hbase20:34681-shortCompactions-0] compactions.Compactor(207): Compacting bc739284553c434bbe2f22067bfb697a, keycount=150, bloomtype=ROW, size=12.5 K, encoding=NONE, compression=NONE, seqNum=302, earliestPutTs=1681527224314 2023-04-15 02:53:49,038 DEBUG [RS:0;jenkins-hbase20:34681-shortCompactions-0] compactions.Compactor(207): Compacting c706a717a77e466ab0d6d968caccf6ce, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=340, earliestPutTs=1681527224456 2023-04-15 02:53:49,038 DEBUG [RS:0;jenkins-hbase20:34681-shortCompactions-0] compactions.Compactor(207): Compacting 42d7ae1915bf4283935d2781d54de6d7, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=352, earliestPutTs=1681527225383 2023-04-15 02:53:49,039 DEBUG [RS:0;jenkins-hbase20:34681-shortCompactions-0] compactions.Compactor(207): Compacting fc51d87e600940bda48e00d50d6afe40, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=390, earliestPutTs=1681527226572 2023-04-15 02:53:49,050 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-15 02:53:49,051 INFO [RS:0;jenkins-hbase20:34681-shortCompactions-0] throttle.PressureAwareThroughputController(145): c44270c3412c22e2c805f466aae6ff56#C#compaction#68 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-15 02:53:49,056 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-15 02:53:49,057 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-15 02:53:49,066 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-15 02:53:49,066 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-15 02:53:49,066 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-15 02:53:49,066 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-15 02:53:49,066 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-15 02:53:49,066 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-15 02:53:49,066 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:53:49,067 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:53:49,067 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-15 02:53:49,067 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-15 02:53:49,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-15 02:53:49,067 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:49,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:53:49,068 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:49,068 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-15 02:53:49,068 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:53:49,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-15 02:53:49,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:53:49,068 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-15 02:53:49,070 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-15 02:53:49,070 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-15 02:53:49,070 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-15 02:53:49,070 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-15 02:53:49,070 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-15 02:53:49,078 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-15 02:53:49,078 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681527213260.c44270c3412c22e2c805f466aae6ff56. 2023-04-15 02:53:49,078 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-15 02:53:49,078 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681527213260.c44270c3412c22e2c805f466aae6ff56. started... 2023-04-15 02:53:49,078 INFO [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing c44270c3412c22e2c805f466aae6ff56 3/3 column families, dataSize=40.25 KB heapSize=120.28 KB 2023-04-15 02:53:49,079 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c44270c3412c22e2c805f466aae6ff56, store=A 2023-04-15 02:53:49,079 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-15 02:53:49,079 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c44270c3412c22e2c805f466aae6ff56, store=B 2023-04-15 02:53:49,079 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-15 02:53:49,079 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c44270c3412c22e2c805f466aae6ff56, store=C 2023-04-15 02:53:49,079 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-15 02:53:49,086 DEBUG [RS:0;jenkins-hbase20:34681-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/c44270c3412c22e2c805f466aae6ff56/.tmp/C/3de6cdf4cc5f47ff9def540e573bb52d as hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/c44270c3412c22e2c805f466aae6ff56/C/3de6cdf4cc5f47ff9def540e573bb52d 2023-04-15 02:53:49,089 INFO [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=13.42 KB at sequenceid=400 (bloomFilter=true), to=hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/c44270c3412c22e2c805f466aae6ff56/.tmp/A/3a38a999ff8441b7b08fcc09067aca06 2023-04-15 02:53:49,093 INFO [RS:0;jenkins-hbase20:34681-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in c44270c3412c22e2c805f466aae6ff56/C of c44270c3412c22e2c805f466aae6ff56 into 3de6cdf4cc5f47ff9def540e573bb52d(size=12.6 K), total size for store is 12.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-15 02:53:49,093 DEBUG [RS:0;jenkins-hbase20:34681-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for c44270c3412c22e2c805f466aae6ff56: 2023-04-15 02:53:49,093 INFO [RS:0;jenkins-hbase20:34681-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681527213260.c44270c3412c22e2c805f466aae6ff56., storeName=c44270c3412c22e2c805f466aae6ff56/C, priority=12, startTime=1681527228967; duration=0sec 2023-04-15 02:53:49,093 DEBUG [RS:0;jenkins-hbase20:34681-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-15 02:53:49,103 INFO [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=13.42 KB at sequenceid=400 (bloomFilter=true), to=hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/c44270c3412c22e2c805f466aae6ff56/.tmp/B/9afaa7d2a9be47d185532e68809bfe92 2023-04-15 02:53:49,117 INFO [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=13.42 KB at sequenceid=400 (bloomFilter=true), to=hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/c44270c3412c22e2c805f466aae6ff56/.tmp/C/f6f90e98a27f4485ba53d503154f7322 2023-04-15 02:53:49,124 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/c44270c3412c22e2c805f466aae6ff56/.tmp/A/3a38a999ff8441b7b08fcc09067aca06 as hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/c44270c3412c22e2c805f466aae6ff56/A/3a38a999ff8441b7b08fcc09067aca06 2023-04-15 02:53:49,129 INFO [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/c44270c3412c22e2c805f466aae6ff56/A/3a38a999ff8441b7b08fcc09067aca06, entries=150, sequenceid=400, filesize=11.9 K 2023-04-15 02:53:49,129 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/c44270c3412c22e2c805f466aae6ff56/.tmp/B/9afaa7d2a9be47d185532e68809bfe92 as hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/c44270c3412c22e2c805f466aae6ff56/B/9afaa7d2a9be47d185532e68809bfe92 2023-04-15 02:53:49,134 INFO [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/c44270c3412c22e2c805f466aae6ff56/B/9afaa7d2a9be47d185532e68809bfe92, entries=150, sequenceid=400, filesize=11.9 K 2023-04-15 02:53:49,135 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/c44270c3412c22e2c805f466aae6ff56/.tmp/C/f6f90e98a27f4485ba53d503154f7322 as hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/c44270c3412c22e2c805f466aae6ff56/C/f6f90e98a27f4485ba53d503154f7322 2023-04-15 02:53:49,140 INFO [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/c44270c3412c22e2c805f466aae6ff56/C/f6f90e98a27f4485ba53d503154f7322, entries=150, sequenceid=400, filesize=11.9 K 2023-04-15 02:53:49,141 INFO [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~40.25 KB/41220, heapSize ~120.23 KB/123120, currentSize=0 B/0 for c44270c3412c22e2c805f466aae6ff56 in 63ms, sequenceid=400, compaction requested=false 2023-04-15 02:53:49,141 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for c44270c3412c22e2c805f466aae6ff56: 2023-04-15 02:53:49,141 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681527213260.c44270c3412c22e2c805f466aae6ff56. 2023-04-15 02:53:49,141 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-15 02:53:49,141 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-15 02:53:49,141 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:49,141 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-15 02:53:49,141 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34681,1681527209133' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-15 02:53:49,143 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:49,143 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:53:49,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:49,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:53:49,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:53:49,143 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:53:49,143 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-15 02:53:49,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:53:49,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:53:49,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:53:49,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:49,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:53:49,145 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34681,1681527209133' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-15 02:53:49,145 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2aa08804[Count = 0] remaining members to acquire global barrier 2023-04-15 02:53:49,145 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-15 02:53:49,145 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:53:49,145 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:53:49,145 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:53:49,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:53:49,145 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-15 02:53:49,145 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-15 02:53:49,145 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34681,1681527209133' in zk 2023-04-15 02:53:49,145 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:49,146 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-15 02:53:49,147 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:49,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:49,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:53:49,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:53:49,147 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-15 02:53:49,147 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-15 02:53:49,147 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-15 02:53:49,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:53:49,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:53:49,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:53:49,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:49,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:53:49,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:53:49,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:49,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34681,1681527209133': 2023-04-15 02:53:49,149 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34681,1681527209133' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-15 02:53:49,149 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-15 02:53:49,149 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-15 02:53:49,149 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-15 02:53:49,149 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-15 02:53:49,149 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-15 02:53:49,150 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:53:49,150 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:53:49,150 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:53:49,150 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-15 02:53:49,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:53:49,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:53:49,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:53:49,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:53:49,151 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-15 02:53:49,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-15 02:53:49,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:53:49,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:53:49,151 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:49,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:53:49,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:53:49,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:53:49,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:49,153 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:49,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:53:49,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:53:49,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:49,154 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:49,154 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-15 02:53:49,154 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:53:49,154 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:53:49,154 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:49,154 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:53:49,154 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:53:49,154 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-15 02:53:49,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-15 02:53:49,154 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-15 02:53:49,154 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-15 02:53:49,154 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-15 02:53:49,154 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:53:49,154 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-15 02:53:49,155 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-15 02:53:49,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-15 02:53:49,155 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-15 02:53:49,155 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-15 02:53:49,255 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-15 02:53:49,255 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-15 02:53:49,257 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-15 02:53:49,258 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-15 02:53:49,258 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-15 02:53:49,258 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-15 02:53:49,258 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-15 02:53:49,258 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-15 02:53:49,258 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-15 02:53:49,259 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:53:49,259 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:53:49,259 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-15 02:53:49,259 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:49,260 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-15 02:53:49,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-15 02:53:49,260 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:49,260 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-15 02:53:49,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:53:49,260 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:53:49,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-15 02:53:49,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:53:49,261 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-15 02:53:49,261 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-15 02:53:49,261 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-15 02:53:49,261 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-15 02:53:49,261 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-15 02:53:49,261 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-15 02:53:49,282 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-15 02:53:49,282 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681527213260.c44270c3412c22e2c805f466aae6ff56. 2023-04-15 02:53:49,282 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-15 02:53:49,282 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681527213260.c44270c3412c22e2c805f466aae6ff56. started... 2023-04-15 02:53:49,282 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for c44270c3412c22e2c805f466aae6ff56: 2023-04-15 02:53:49,282 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681527213260.c44270c3412c22e2c805f466aae6ff56. 2023-04-15 02:53:49,282 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-15 02:53:49,282 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-15 02:53:49,282 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:49,282 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-15 02:53:49,282 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34681,1681527209133' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-15 02:53:49,285 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:49,285 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:53:49,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:49,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:53:49,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:53:49,286 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:53:49,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:53:49,286 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-15 02:53:49,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:53:49,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:53:49,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:49,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:53:49,287 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34681,1681527209133' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-15 02:53:49,287 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4475f330[Count = 0] remaining members to acquire global barrier 2023-04-15 02:53:49,287 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-15 02:53:49,287 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:53:49,288 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:53:49,288 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:53:49,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:53:49,288 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-15 02:53:49,288 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:49,288 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-15 02:53:49,288 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-15 02:53:49,288 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34681,1681527209133' in zk 2023-04-15 02:53:49,289 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:49,289 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-15 02:53:49,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:49,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:53:49,289 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-15 02:53:49,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:53:49,289 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-15 02:53:49,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:53:49,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:53:49,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:53:49,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:49,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:53:49,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:53:49,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:49,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34681,1681527209133': 2023-04-15 02:53:49,291 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34681,1681527209133' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-15 02:53:49,291 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-15 02:53:49,291 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-15 02:53:49,291 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-15 02:53:49,291 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-15 02:53:49,291 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-15 02:53:49,292 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:53:49,292 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:53:49,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:53:49,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:53:49,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:53:49,292 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:53:49,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:53:49,292 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-15 02:53:49,292 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-15 02:53:49,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-15 02:53:49,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:53:49,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:53:49,293 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:49,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:53:49,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:53:49,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:53:49,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:49,294 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:49,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:53:49,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:53:49,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:49,295 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:49,295 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-15 02:53:49,295 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:53:49,295 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-15 02:53:49,295 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-15 02:53:49,295 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:53:49,295 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-15 02:53:49,295 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-15 02:53:49,295 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-15 02:53:49,295 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-15 02:53:49,295 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:49,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-15 02:53:49,296 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-15 02:53:49,296 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:53:49,296 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:53:49,296 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:53:49,296 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-15 02:53:49,296 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-15 02:53:49,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-15 02:53:49,396 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-15 02:53:49,396 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-15 02:53:49,398 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-15 02:53:49,400 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-15 02:53:49,400 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-15 02:53:49,400 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-15 02:53:49,400 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-15 02:53:49,400 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-15 02:53:49,400 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-15 02:53:49,400 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:53:49,400 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:53:49,401 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:49,401 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-15 02:53:49,401 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-15 02:53:49,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-15 02:53:49,401 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:49,401 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-15 02:53:49,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:53:49,401 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:53:49,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-15 02:53:49,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:53:49,402 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-15 02:53:49,402 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-15 02:53:49,402 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-15 02:53:49,402 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-15 02:53:49,402 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-15 02:53:49,402 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-15 02:53:49,403 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-15 02:53:49,403 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-15 02:53:49,403 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681527213260.c44270c3412c22e2c805f466aae6ff56. 2023-04-15 02:53:49,403 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681527213260.c44270c3412c22e2c805f466aae6ff56. started... 2023-04-15 02:53:49,403 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for c44270c3412c22e2c805f466aae6ff56: 2023-04-15 02:53:49,403 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681527213260.c44270c3412c22e2c805f466aae6ff56. 2023-04-15 02:53:49,403 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-15 02:53:49,403 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-15 02:53:49,403 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:49,403 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-15 02:53:49,403 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34681,1681527209133' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-15 02:53:49,404 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:49,404 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:53:49,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:49,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:53:49,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:53:49,405 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:53:49,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:53:49,405 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-15 02:53:49,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:53:49,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:53:49,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:49,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:53:49,406 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34681,1681527209133' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-15 02:53:49,406 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@56ce5eed[Count = 0] remaining members to acquire global barrier 2023-04-15 02:53:49,406 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-15 02:53:49,406 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:53:49,407 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:53:49,407 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:53:49,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:53:49,407 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-15 02:53:49,407 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-15 02:53:49,407 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34681,1681527209133' in zk 2023-04-15 02:53:49,407 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:49,407 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-15 02:53:49,407 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:49,407 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-15 02:53:49,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:49,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:53:49,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:53:49,408 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-15 02:53:49,408 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-15 02:53:49,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:53:49,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:53:49,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:53:49,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:49,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:53:49,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:53:49,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:49,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34681,1681527209133': 2023-04-15 02:53:49,412 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34681,1681527209133' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-15 02:53:49,412 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-15 02:53:49,412 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-15 02:53:49,412 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-15 02:53:49,412 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-15 02:53:49,412 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-15 02:53:49,413 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:53:49,414 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:53:49,414 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-15 02:53:49,414 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:53:49,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:53:49,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:53:49,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:53:49,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:53:49,415 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-15 02:53:49,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-15 02:53:49,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:53:49,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:53:49,416 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:49,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:53:49,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:53:49,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:53:49,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:49,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:53:49,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:53:49,419 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:49,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:49,420 DEBUG [RS:0;jenkins-hbase20:34681-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/c44270c3412c22e2c805f466aae6ff56/.tmp/B/fcd3cf354e5140479901c13ba09dae61 as hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/c44270c3412c22e2c805f466aae6ff56/B/fcd3cf354e5140479901c13ba09dae61 2023-04-15 02:53:49,427 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-15 02:53:49,427 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-15 02:53:49,427 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-15 02:53:49,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-15 02:53:49,431 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:49,431 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-15 02:53:49,432 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-15 02:53:49,431 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-15 02:53:49,432 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-15 02:53:49,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-15 02:53:49,432 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:53:49,432 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:53:49,432 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-15 02:53:49,432 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-15 02:53:49,432 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:49,432 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:53:49,432 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:53:49,432 INFO [RS:0;jenkins-hbase20:34681-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in c44270c3412c22e2c805f466aae6ff56/B of c44270c3412c22e2c805f466aae6ff56 into fcd3cf354e5140479901c13ba09dae61(size=12.6 K), total size for store is 24.5 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-15 02:53:49,432 DEBUG [RS:0;jenkins-hbase20:34681-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for c44270c3412c22e2c805f466aae6ff56: 2023-04-15 02:53:49,432 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:53:49,432 INFO [RS:0;jenkins-hbase20:34681-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681527213260.c44270c3412c22e2c805f466aae6ff56., storeName=c44270c3412c22e2c805f466aae6ff56/B, priority=12, startTime=1681527228967; duration=0sec 2023-04-15 02:53:49,432 DEBUG [RS:0;jenkins-hbase20:34681-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-15 02:53:49,447 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34681] regionserver.HRegion(9158): Flush requested on c44270c3412c22e2c805f466aae6ff56 2023-04-15 02:53:49,447 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing c44270c3412c22e2c805f466aae6ff56 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-15 02:53:49,447 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c44270c3412c22e2c805f466aae6ff56, store=A 2023-04-15 02:53:49,447 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-15 02:53:49,447 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c44270c3412c22e2c805f466aae6ff56, store=B 2023-04-15 02:53:49,447 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-15 02:53:49,447 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c44270c3412c22e2c805f466aae6ff56, store=C 2023-04-15 02:53:49,448 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-15 02:53:49,510 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681527229496, max timestamp=1681527229508Num uniques 859; 2023-04-15 02:53:49,510 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-2] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-15 02:53:49,510 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681527229496, max timestamp=1681527229508Num uniques 859; 2023-04-15 02:53:49,510 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-7] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-15 02:53:49,532 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-15 02:53:49,533 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-15 02:53:49,535 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-15 02:53:49,536 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-15 02:53:49,536 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-15 02:53:49,536 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-15 02:53:49,536 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-15 02:53:49,536 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-15 02:53:49,536 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-15 02:53:49,537 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:53:49,537 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:53:49,538 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-15 02:53:49,538 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:49,538 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-15 02:53:49,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-15 02:53:49,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:53:49,539 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:53:49,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-15 02:53:49,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:53:49,540 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-15 02:53:49,540 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-15 02:53:49,540 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-15 02:53:49,540 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:49,540 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-15 02:53:49,540 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-15 02:53:49,540 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-15 02:53:49,540 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-15 02:53:49,545 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c44270c3412c22e2c805f466aae6ff56, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:53:49,545 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34681] ipc.CallRunner(144): callId: 133 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:38504 deadline: 1681527289543, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c44270c3412c22e2c805f466aae6ff56, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:49,551 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-15 02:53:49,551 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-15 02:53:49,551 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681527213260.c44270c3412c22e2c805f466aae6ff56. 2023-04-15 02:53:49,551 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681527213260.c44270c3412c22e2c805f466aae6ff56. started... 2023-04-15 02:53:49,552 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681527213260.c44270c3412c22e2c805f466aae6ff56. as already flushing 2023-04-15 02:53:49,552 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681527213260.c44270c3412c22e2c805f466aae6ff56. 2023-04-15 02:53:49,552 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-15 02:53:49,552 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-15 02:53:49,552 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:49,552 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-15 02:53:49,552 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34681,1681527209133' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-15 02:53:49,553 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:49,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:49,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:53:49,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:53:49,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:53:49,557 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:53:49,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:53:49,557 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:53:49,557 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-15 02:53:49,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:53:49,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:49,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:53:49,558 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34681,1681527209133' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-15 02:53:49,558 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-15 02:53:49,558 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2276b779[Count = 0] remaining members to acquire global barrier 2023-04-15 02:53:49,558 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:53:49,559 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:53:49,559 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:53:49,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:53:49,559 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-15 02:53:49,559 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-15 02:53:49,559 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34681,1681527209133' in zk 2023-04-15 02:53:49,559 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:49,559 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-15 02:53:49,560 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-15 02:53:49,560 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-15 02:53:49,560 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-15 02:53:49,560 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:49,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:49,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:53:49,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:53:49,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:53:49,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:53:49,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:53:49,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:49,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:53:49,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:53:49,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:49,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34681,1681527209133': 2023-04-15 02:53:49,564 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34681,1681527209133' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-15 02:53:49,564 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-15 02:53:49,564 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-15 02:53:49,564 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-15 02:53:49,564 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-15 02:53:49,564 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-15 02:53:49,565 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:53:49,565 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:53:49,565 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-15 02:53:49,565 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:53:49,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:53:49,566 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-15 02:53:49,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-15 02:53:49,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:53:49,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:53:49,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:53:49,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:53:49,566 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:49,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:53:49,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:53:49,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:53:49,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:53:49,568 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:49,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:49,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:53:49,570 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-15 02:53:49,570 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-15 02:53:49,570 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:49,570 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:53:49,570 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:53:49,570 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-15 02:53:49,570 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-15 02:53:49,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-15 02:53:49,570 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-15 02:53:49,570 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-15 02:53:49,570 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:49,571 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:53:49,571 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:53:49,571 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-15 02:53:49,571 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-15 02:53:49,571 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:53:49,571 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-15 02:53:49,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-15 02:53:49,574 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-15 02:53:49,648 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c44270c3412c22e2c805f466aae6ff56, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:53:49,648 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34681] ipc.CallRunner(144): callId: 135 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:38504 deadline: 1681527289646, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c44270c3412c22e2c805f466aae6ff56, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:49,671 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-15 02:53:49,671 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-15 02:53:49,673 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-15 02:53:49,674 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-15 02:53:49,674 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-15 02:53:49,674 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-15 02:53:49,674 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-15 02:53:49,674 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-15 02:53:49,674 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-15 02:53:49,674 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:53:49,674 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:53:49,675 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-15 02:53:49,675 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:49,675 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-15 02:53:49,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-15 02:53:49,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:53:49,675 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:49,675 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-15 02:53:49,676 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:53:49,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-15 02:53:49,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:53:49,676 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-15 02:53:49,676 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-15 02:53:49,676 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-15 02:53:49,676 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-15 02:53:49,676 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-15 02:53:49,677 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-15 02:53:49,677 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-15 02:53:49,677 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681527213260.c44270c3412c22e2c805f466aae6ff56. 2023-04-15 02:53:49,677 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681527213260.c44270c3412c22e2c805f466aae6ff56. started... 2023-04-15 02:53:49,677 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681527213260.c44270c3412c22e2c805f466aae6ff56. as already flushing 2023-04-15 02:53:49,677 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681527213260.c44270c3412c22e2c805f466aae6ff56. 2023-04-15 02:53:49,677 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-15 02:53:49,677 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-15 02:53:49,677 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:49,677 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-15 02:53:49,677 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34681,1681527209133' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-15 02:53:49,678 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:49,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:49,678 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:53:49,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:53:49,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:53:49,679 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:53:49,679 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-15 02:53:49,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:53:49,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:53:49,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:53:49,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:49,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:53:49,680 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34681,1681527209133' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-15 02:53:49,680 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-15 02:53:49,680 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@c38fb9b[Count = 0] remaining members to acquire global barrier 2023-04-15 02:53:49,680 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:53:49,685 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:53:49,685 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:53:49,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:53:49,685 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-15 02:53:49,685 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-15 02:53:49,685 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34681,1681527209133' in zk 2023-04-15 02:53:49,687 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:49,687 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-15 02:53:49,687 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-15 02:53:49,688 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-15 02:53:49,688 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:49,688 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-15 02:53:49,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:49,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:53:49,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:53:49,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:53:49,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:53:49,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:53:49,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:49,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:53:49,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:53:49,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:49,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34681,1681527209133': 2023-04-15 02:53:49,690 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34681,1681527209133' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-15 02:53:49,690 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-15 02:53:49,690 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-15 02:53:49,690 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-15 02:53:49,690 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-15 02:53:49,690 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-15 02:53:49,691 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:53:49,691 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:53:49,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:53:49,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:53:49,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:53:49,691 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:53:49,691 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-15 02:53:49,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:53:49,692 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-15 02:53:49,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-15 02:53:49,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:53:49,692 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:49,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:53:49,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:53:49,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:53:49,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:53:49,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:49,693 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:49,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:53:49,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:53:49,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:49,694 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:49,694 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:53:49,694 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:53:49,694 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-15 02:53:49,694 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-15 02:53:49,694 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-15 02:53:49,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-15 02:53:49,695 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-15 02:53:49,695 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-15 02:53:49,695 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-15 02:53:49,695 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-15 02:53:49,695 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-15 02:53:49,695 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-15 02:53:49,695 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:49,695 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:53:49,695 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:53:49,695 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:53:49,695 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-15 02:53:49,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-15 02:53:49,795 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-15 02:53:49,796 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-15 02:53:49,797 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-15 02:53:49,798 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-15 02:53:49,798 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-15 02:53:49,798 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-15 02:53:49,798 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-15 02:53:49,798 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-15 02:53:49,798 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-15 02:53:49,799 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:53:49,799 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:53:49,799 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-15 02:53:49,799 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:49,799 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-15 02:53:49,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-15 02:53:49,800 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:49,800 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-15 02:53:49,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:53:49,800 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:53:49,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-15 02:53:49,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:53:49,800 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-15 02:53:49,800 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-15 02:53:49,800 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-15 02:53:49,801 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-15 02:53:49,801 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-15 02:53:49,801 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-15 02:53:49,802 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-15 02:53:49,802 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-15 02:53:49,802 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681527213260.c44270c3412c22e2c805f466aae6ff56. 2023-04-15 02:53:49,802 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681527213260.c44270c3412c22e2c805f466aae6ff56. started... 2023-04-15 02:53:49,802 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681527213260.c44270c3412c22e2c805f466aae6ff56. as already flushing 2023-04-15 02:53:49,802 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681527213260.c44270c3412c22e2c805f466aae6ff56. 2023-04-15 02:53:49,802 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-15 02:53:49,802 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-15 02:53:49,802 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:49,802 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-15 02:53:49,802 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34681,1681527209133' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-15 02:53:49,803 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:49,803 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:53:49,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:49,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:53:49,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:53:49,803 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:53:49,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:53:49,803 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-15 02:53:49,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:53:49,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:53:49,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:49,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:53:49,804 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34681,1681527209133' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-15 02:53:49,804 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@673f755f[Count = 0] remaining members to acquire global barrier 2023-04-15 02:53:49,804 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-15 02:53:49,804 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:53:49,805 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:53:49,805 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:53:49,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:53:49,805 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-15 02:53:49,805 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-15 02:53:49,805 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34681,1681527209133' in zk 2023-04-15 02:53:49,805 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:49,805 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-15 02:53:49,807 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-15 02:53:49,807 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:49,807 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-15 02:53:49,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:49,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:53:49,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:53:49,807 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-15 02:53:49,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:53:49,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:53:49,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:53:49,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:49,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:53:49,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:53:49,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:49,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34681,1681527209133': 2023-04-15 02:53:49,809 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34681,1681527209133' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-15 02:53:49,809 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-15 02:53:49,809 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-15 02:53:49,809 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-15 02:53:49,809 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-15 02:53:49,809 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-15 02:53:49,810 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:53:49,810 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:53:49,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:53:49,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:53:49,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:53:49,810 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-15 02:53:49,811 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:53:49,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:53:49,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:53:49,811 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:49,811 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-15 02:53:49,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-15 02:53:49,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:53:49,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:53:49,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:53:49,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:53:49,812 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:49,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:49,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:53:49,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:53:49,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:49,813 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:49,813 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:53:49,813 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:53:49,813 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-15 02:53:49,813 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-15 02:53:49,814 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-15 02:53:49,814 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-15 02:53:49,814 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-15 02:53:49,813 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:49,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-15 02:53:49,814 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-15 02:53:49,814 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-15 02:53:49,814 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:53:49,814 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:53:49,814 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:53:49,814 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-15 02:53:49,814 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-15 02:53:49,814 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-15 02:53:49,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-15 02:53:49,832 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c44270c3412c22e2c805f466aae6ff56, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:53:49,833 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34681] ipc.CallRunner(144): callId: 142 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:38542 deadline: 1681527289830, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c44270c3412c22e2c805f466aae6ff56, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:49,841 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c44270c3412c22e2c805f466aae6ff56, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:53:49,841 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34681] ipc.CallRunner(144): callId: 147 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:38520 deadline: 1681527289839, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c44270c3412c22e2c805f466aae6ff56, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:49,853 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c44270c3412c22e2c805f466aae6ff56, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:53:49,853 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34681] ipc.CallRunner(144): callId: 137 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:38504 deadline: 1681527289849, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c44270c3412c22e2c805f466aae6ff56, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:49,860 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=414 (bloomFilter=true), to=hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/c44270c3412c22e2c805f466aae6ff56/.tmp/A/9adad4a9679d4598acc5a03552f1531e 2023-04-15 02:53:49,914 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-15 02:53:49,915 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-15 02:53:49,916 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-15 02:53:49,917 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-15 02:53:49,917 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-15 02:53:49,917 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-15 02:53:49,917 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-15 02:53:49,917 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-15 02:53:49,917 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-15 02:53:49,918 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:53:49,918 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:53:49,918 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:49,918 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-15 02:53:49,918 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-15 02:53:49,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-15 02:53:49,919 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:49,919 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-15 02:53:49,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:53:49,919 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:53:49,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-15 02:53:49,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:53:49,919 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-15 02:53:49,920 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-15 02:53:49,920 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-15 02:53:49,920 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-15 02:53:49,920 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-15 02:53:49,920 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-15 02:53:49,920 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-15 02:53:49,920 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-15 02:53:49,920 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681527213260.c44270c3412c22e2c805f466aae6ff56. 2023-04-15 02:53:49,920 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681527213260.c44270c3412c22e2c805f466aae6ff56. started... 2023-04-15 02:53:49,920 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681527213260.c44270c3412c22e2c805f466aae6ff56. as already flushing 2023-04-15 02:53:49,920 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681527213260.c44270c3412c22e2c805f466aae6ff56. 2023-04-15 02:53:49,920 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-15 02:53:49,920 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-15 02:53:49,920 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:49,921 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-15 02:53:49,921 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34681,1681527209133' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-15 02:53:49,922 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:49,922 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:53:49,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:49,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:53:49,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:53:49,922 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:53:49,922 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-15 02:53:49,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:53:49,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:53:49,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:53:49,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:49,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:53:49,924 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34681,1681527209133' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-15 02:53:49,924 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@43227663[Count = 0] remaining members to acquire global barrier 2023-04-15 02:53:49,924 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-15 02:53:49,924 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:53:49,924 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:53:49,924 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:53:49,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:53:49,924 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-15 02:53:49,924 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:49,925 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-15 02:53:49,924 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-15 02:53:49,925 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34681,1681527209133' in zk 2023-04-15 02:53:49,925 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:49,925 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-15 02:53:49,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:49,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:53:49,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:53:49,925 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-15 02:53:49,925 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-15 02:53:49,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:53:49,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:53:49,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:53:49,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:49,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:53:49,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:53:49,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:49,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34681,1681527209133': 2023-04-15 02:53:49,927 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34681,1681527209133' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-15 02:53:49,927 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-15 02:53:49,927 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-15 02:53:49,927 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-15 02:53:49,927 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-15 02:53:49,927 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-15 02:53:49,928 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:53:49,928 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:53:49,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:53:49,928 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-15 02:53:49,928 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:53:49,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:53:49,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:53:49,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:53:49,928 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-15 02:53:49,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-15 02:53:49,929 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:49,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:53:49,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:53:49,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:53:49,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:53:49,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:53:49,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:49,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:53:49,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:53:49,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:49,934 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:49,935 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:49,936 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-15 02:53:49,936 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:53:49,936 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-15 02:53:49,936 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-15 02:53:49,936 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-15 02:53:49,936 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-15 02:53:49,936 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-15 02:53:49,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-15 02:53:49,936 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-15 02:53:49,936 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:53:49,936 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-15 02:53:49,936 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:49,936 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:53:49,936 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:53:49,936 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:53:49,936 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-15 02:53:49,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-15 02:53:50,036 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-15 02:53:50,037 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-15 02:53:50,038 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-15 02:53:50,039 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-15 02:53:50,039 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-15 02:53:50,039 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-15 02:53:50,039 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-15 02:53:50,039 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-15 02:53:50,039 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-15 02:53:50,039 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:53:50,039 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:53:50,040 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:50,040 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-15 02:53:50,040 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-15 02:53:50,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-15 02:53:50,040 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:50,040 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-15 02:53:50,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:53:50,041 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:53:50,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-15 02:53:50,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:53:50,041 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-15 02:53:50,041 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-15 02:53:50,041 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-15 02:53:50,041 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-15 02:53:50,041 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-15 02:53:50,042 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-15 02:53:50,042 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-15 02:53:50,042 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-15 02:53:50,042 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681527213260.c44270c3412c22e2c805f466aae6ff56. 2023-04-15 02:53:50,042 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681527213260.c44270c3412c22e2c805f466aae6ff56. started... 2023-04-15 02:53:50,042 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681527213260.c44270c3412c22e2c805f466aae6ff56. as already flushing 2023-04-15 02:53:50,042 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681527213260.c44270c3412c22e2c805f466aae6ff56. 2023-04-15 02:53:50,042 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-15 02:53:50,042 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-15 02:53:50,042 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:50,042 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-15 02:53:50,042 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34681,1681527209133' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-15 02:53:50,043 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:50,043 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:53:50,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:50,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:53:50,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:53:50,043 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:53:50,043 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-15 02:53:50,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:53:50,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:53:50,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:53:50,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:50,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:53:50,044 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34681,1681527209133' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-15 02:53:50,044 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-15 02:53:50,044 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5478cc01[Count = 0] remaining members to acquire global barrier 2023-04-15 02:53:50,044 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:53:50,045 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:53:50,045 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:53:50,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:53:50,045 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-15 02:53:50,045 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-15 02:53:50,045 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34681,1681527209133' in zk 2023-04-15 02:53:50,045 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:50,045 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-15 02:53:50,046 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:50,046 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-15 02:53:50,047 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-15 02:53:50,047 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-15 02:53:50,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:50,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:53:50,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:53:50,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:53:50,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:53:50,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:53:50,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:50,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:53:50,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:53:50,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:50,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34681,1681527209133': 2023-04-15 02:53:50,049 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34681,1681527209133' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-15 02:53:50,049 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-15 02:53:50,049 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-15 02:53:50,049 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-15 02:53:50,049 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-15 02:53:50,049 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-15 02:53:50,050 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:53:50,050 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:53:50,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:53:50,050 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:53:50,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:53:50,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:53:50,050 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-15 02:53:50,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:53:50,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:53:50,050 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-15 02:53:50,050 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:50,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-15 02:53:50,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:53:50,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:53:50,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:53:50,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:53:50,051 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:50,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:50,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:53:50,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:53:50,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:50,053 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:50,053 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:53:50,053 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:53:50,053 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-15 02:53:50,053 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-15 02:53:50,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-15 02:53:50,053 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-15 02:53:50,053 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:50,053 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:53:50,053 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:53:50,053 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:53:50,053 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-15 02:53:50,053 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-15 02:53:50,053 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-15 02:53:50,054 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-15 02:53:50,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-15 02:53:50,054 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-15 02:53:50,054 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-15 02:53:50,054 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-15 02:53:50,154 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-15 02:53:50,154 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-15 02:53:50,156 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-15 02:53:50,156 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-15 02:53:50,156 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c44270c3412c22e2c805f466aae6ff56, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:53:50,156 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-15 02:53:50,156 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34681] ipc.CallRunner(144): callId: 139 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:38504 deadline: 1681527290155, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c44270c3412c22e2c805f466aae6ff56, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:50,156 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-15 02:53:50,156 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-15 02:53:50,157 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-15 02:53:50,157 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-15 02:53:50,157 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:53:50,157 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:53:50,158 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-15 02:53:50,158 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:50,158 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-15 02:53:50,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-15 02:53:50,158 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:50,158 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-15 02:53:50,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:53:50,159 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:53:50,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-15 02:53:50,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:53:50,159 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-15 02:53:50,159 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-15 02:53:50,159 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-15 02:53:50,159 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-15 02:53:50,159 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-15 02:53:50,159 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-15 02:53:50,160 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-15 02:53:50,160 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-15 02:53:50,160 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681527213260.c44270c3412c22e2c805f466aae6ff56. 2023-04-15 02:53:50,160 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681527213260.c44270c3412c22e2c805f466aae6ff56. started... 2023-04-15 02:53:50,160 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681527213260.c44270c3412c22e2c805f466aae6ff56. as already flushing 2023-04-15 02:53:50,160 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681527213260.c44270c3412c22e2c805f466aae6ff56. 2023-04-15 02:53:50,160 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-15 02:53:50,160 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-15 02:53:50,160 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:50,160 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-15 02:53:50,160 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34681,1681527209133' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-15 02:53:50,161 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:50,161 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:53:50,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:50,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:53:50,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:53:50,161 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:53:50,161 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-15 02:53:50,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:53:50,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:53:50,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:53:50,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:50,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:53:50,162 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34681,1681527209133' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-15 02:53:50,162 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-15 02:53:50,162 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1fd71136[Count = 0] remaining members to acquire global barrier 2023-04-15 02:53:50,162 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:53:50,163 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:53:50,163 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:53:50,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:53:50,163 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-15 02:53:50,163 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-15 02:53:50,163 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:50,163 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-15 02:53:50,163 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34681,1681527209133' in zk 2023-04-15 02:53:50,164 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:50,164 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-15 02:53:50,164 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-15 02:53:50,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:50,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:53:50,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:53:50,164 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-15 02:53:50,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:53:50,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:53:50,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:53:50,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:50,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:53:50,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:53:50,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:50,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34681,1681527209133': 2023-04-15 02:53:50,166 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34681,1681527209133' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-15 02:53:50,166 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-15 02:53:50,166 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-15 02:53:50,166 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-15 02:53:50,166 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-15 02:53:50,166 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-15 02:53:50,167 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:53:50,167 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-15 02:53:50,167 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:53:50,167 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:53:50,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:53:50,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:53:50,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:53:50,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:53:50,167 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-15 02:53:50,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-15 02:53:50,167 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:50,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:53:50,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:53:50,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:53:50,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:53:50,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:53:50,168 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:50,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:50,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:53:50,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:53:50,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:50,171 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:50,171 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:53:50,171 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-15 02:53:50,171 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-15 02:53:50,171 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-15 02:53:50,171 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-15 02:53:50,171 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-15 02:53:50,171 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:53:50,171 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:50,171 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:53:50,171 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:53:50,171 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:53:50,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-15 02:53:50,171 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-15 02:53:50,171 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-15 02:53:50,171 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-15 02:53:50,172 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-15 02:53:50,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-15 02:53:50,271 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-15 02:53:50,272 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-15 02:53:50,273 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-15 02:53:50,274 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=414 (bloomFilter=true), to=hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/c44270c3412c22e2c805f466aae6ff56/.tmp/B/7cec6ef3d3de46b88188034a7cdb0313 2023-04-15 02:53:50,274 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-15 02:53:50,274 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-15 02:53:50,274 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-15 02:53:50,274 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-15 02:53:50,274 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-15 02:53:50,275 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-15 02:53:50,275 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:53:50,275 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:53:50,275 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-15 02:53:50,275 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:50,276 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-15 02:53:50,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-15 02:53:50,276 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:50,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:53:50,276 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-15 02:53:50,276 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:53:50,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-15 02:53:50,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:53:50,276 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-15 02:53:50,277 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-15 02:53:50,277 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-15 02:53:50,277 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-15 02:53:50,277 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-15 02:53:50,277 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-15 02:53:50,277 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-15 02:53:50,277 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-15 02:53:50,278 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681527213260.c44270c3412c22e2c805f466aae6ff56. 2023-04-15 02:53:50,278 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681527213260.c44270c3412c22e2c805f466aae6ff56. started... 2023-04-15 02:53:50,278 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681527213260.c44270c3412c22e2c805f466aae6ff56. as already flushing 2023-04-15 02:53:50,278 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681527213260.c44270c3412c22e2c805f466aae6ff56. 2023-04-15 02:53:50,278 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-15 02:53:50,278 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-15 02:53:50,278 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:50,278 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-15 02:53:50,278 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34681,1681527209133' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-15 02:53:50,279 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:50,279 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:53:50,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:50,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:53:50,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:53:50,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:53:50,281 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:53:50,281 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-15 02:53:50,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:53:50,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:53:50,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:50,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:53:50,282 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34681,1681527209133' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-15 02:53:50,282 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3de1a1e[Count = 0] remaining members to acquire global barrier 2023-04-15 02:53:50,282 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-15 02:53:50,282 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:53:50,283 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:53:50,283 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:53:50,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:53:50,283 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-15 02:53:50,283 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-15 02:53:50,283 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34681,1681527209133' in zk 2023-04-15 02:53:50,283 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:50,283 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-15 02:53:50,284 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:50,284 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-15 02:53:50,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:50,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:53:50,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:53:50,284 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-15 02:53:50,285 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-15 02:53:50,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:53:50,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:53:50,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:53:50,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:50,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:53:50,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:53:50,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:50,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34681,1681527209133': 2023-04-15 02:53:50,287 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34681,1681527209133' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-15 02:53:50,287 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-15 02:53:50,287 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-15 02:53:50,287 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-15 02:53:50,287 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-15 02:53:50,287 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-15 02:53:50,288 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:53:50,288 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:53:50,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:53:50,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:53:50,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:53:50,288 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:53:50,288 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-15 02:53:50,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:53:50,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:53:50,289 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-15 02:53:50,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-15 02:53:50,289 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:50,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:53:50,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:53:50,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:53:50,290 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:50,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:53:50,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:50,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:53:50,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:53:50,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:50,292 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-15 02:53:50,292 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-15 02:53:50,292 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:50,292 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-15 02:53:50,292 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-15 02:53:50,292 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-15 02:53:50,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-15 02:53:50,292 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-15 02:53:50,292 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:53:50,292 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:53:50,292 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-15 02:53:50,292 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:50,292 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:53:50,292 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-15 02:53:50,292 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-15 02:53:50,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-15 02:53:50,292 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:53:50,292 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-15 02:53:50,292 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:53:50,393 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-15 02:53:50,393 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-15 02:53:50,395 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-15 02:53:50,395 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-15 02:53:50,395 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-15 02:53:50,395 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-15 02:53:50,395 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-15 02:53:50,396 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-15 02:53:50,396 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-15 02:53:50,396 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:53:50,396 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:53:50,397 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-15 02:53:50,397 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:50,397 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-15 02:53:50,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-15 02:53:50,397 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:50,397 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-15 02:53:50,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:53:50,397 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:53:50,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-15 02:53:50,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:53:50,397 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-15 02:53:50,398 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-15 02:53:50,398 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-15 02:53:50,398 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-15 02:53:50,398 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-15 02:53:50,398 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-15 02:53:50,398 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-15 02:53:50,398 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-15 02:53:50,398 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681527213260.c44270c3412c22e2c805f466aae6ff56. 2023-04-15 02:53:50,398 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681527213260.c44270c3412c22e2c805f466aae6ff56. started... 2023-04-15 02:53:50,398 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681527213260.c44270c3412c22e2c805f466aae6ff56. as already flushing 2023-04-15 02:53:50,398 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681527213260.c44270c3412c22e2c805f466aae6ff56. 2023-04-15 02:53:50,398 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-15 02:53:50,398 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-15 02:53:50,398 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:50,398 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-15 02:53:50,398 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34681,1681527209133' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-15 02:53:50,399 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:50,399 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:53:50,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:50,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:53:50,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:53:50,399 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:53:50,399 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-15 02:53:50,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:53:50,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:53:50,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:53:50,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:50,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:53:50,400 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34681,1681527209133' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-15 02:53:50,400 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@531fe568[Count = 0] remaining members to acquire global barrier 2023-04-15 02:53:50,400 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-15 02:53:50,400 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:53:50,401 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:53:50,401 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:53:50,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:53:50,401 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-15 02:53:50,401 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-15 02:53:50,401 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34681,1681527209133' in zk 2023-04-15 02:53:50,401 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:50,401 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-15 02:53:50,402 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:50,402 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-15 02:53:50,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:50,402 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-15 02:53:50,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:53:50,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:53:50,402 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-15 02:53:50,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:53:50,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:53:50,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:53:50,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:50,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:53:50,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:53:50,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:50,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34681,1681527209133': 2023-04-15 02:53:50,404 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34681,1681527209133' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-15 02:53:50,404 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-15 02:53:50,404 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-15 02:53:50,404 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-15 02:53:50,404 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-15 02:53:50,404 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-15 02:53:50,404 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:53:50,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:53:50,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:53:50,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:53:50,405 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:53:50,405 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-15 02:53:50,405 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:53:50,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:53:50,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:53:50,405 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:50,405 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-15 02:53:50,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-15 02:53:50,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:53:50,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:53:50,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:53:50,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:53:50,406 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:50,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:50,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:53:50,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:53:50,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:50,407 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:50,407 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:53:50,407 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-15 02:53:50,408 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-15 02:53:50,408 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-15 02:53:50,408 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-15 02:53:50,408 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-15 02:53:50,408 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-15 02:53:50,407 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:53:50,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-15 02:53:50,408 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-15 02:53:50,408 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-15 02:53:50,408 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:50,408 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-15 02:53:50,408 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:53:50,408 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-15 02:53:50,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-15 02:53:50,408 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:53:50,408 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:53:50,508 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-15 02:53:50,509 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-15 02:53:50,510 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-15 02:53:50,511 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-15 02:53:50,511 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-15 02:53:50,511 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-15 02:53:50,511 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-15 02:53:50,511 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-15 02:53:50,511 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-15 02:53:50,511 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:53:50,511 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:53:50,512 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-15 02:53:50,512 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:50,512 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-15 02:53:50,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-15 02:53:50,512 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:50,512 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-15 02:53:50,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:53:50,513 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:53:50,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-15 02:53:50,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:53:50,513 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-15 02:53:50,513 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-15 02:53:50,513 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-15 02:53:50,513 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-15 02:53:50,513 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-15 02:53:50,513 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-15 02:53:50,513 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-15 02:53:50,513 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-15 02:53:50,513 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681527213260.c44270c3412c22e2c805f466aae6ff56. 2023-04-15 02:53:50,513 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681527213260.c44270c3412c22e2c805f466aae6ff56. started... 2023-04-15 02:53:50,513 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681527213260.c44270c3412c22e2c805f466aae6ff56. as already flushing 2023-04-15 02:53:50,513 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681527213260.c44270c3412c22e2c805f466aae6ff56. 2023-04-15 02:53:50,513 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-15 02:53:50,513 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-15 02:53:50,513 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:50,513 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-15 02:53:50,513 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34681,1681527209133' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-15 02:53:50,514 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:53:50,514 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:50,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:50,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:53:50,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:53:50,515 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:53:50,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:53:50,515 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-15 02:53:50,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:53:50,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:53:50,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:50,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:53:50,515 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34681,1681527209133' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-15 02:53:50,515 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-15 02:53:50,515 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@411533ce[Count = 0] remaining members to acquire global barrier 2023-04-15 02:53:50,516 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:53:50,516 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:53:50,516 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:53:50,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:53:50,516 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-15 02:53:50,516 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-15 02:53:50,516 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34681,1681527209133' in zk 2023-04-15 02:53:50,516 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:50,516 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-15 02:53:50,517 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:50,517 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-15 02:53:50,517 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-15 02:53:50,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:50,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:53:50,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:53:50,517 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-15 02:53:50,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:53:50,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:53:50,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:53:50,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:50,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:53:50,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:53:50,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:50,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34681,1681527209133': 2023-04-15 02:53:50,519 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34681,1681527209133' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-15 02:53:50,519 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-15 02:53:50,519 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-15 02:53:50,519 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-15 02:53:50,519 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-15 02:53:50,519 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-15 02:53:50,519 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:53:50,519 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:53:50,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:53:50,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:53:50,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:53:50,519 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:53:50,519 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-15 02:53:50,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:53:50,520 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-15 02:53:50,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-15 02:53:50,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:53:50,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:53:50,520 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:50,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:53:50,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:53:50,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:53:50,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:50,521 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:50,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:53:50,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:53:50,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:50,523 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:50,523 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-15 02:53:50,523 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:53:50,523 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-15 02:53:50,523 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-15 02:53:50,523 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-15 02:53:50,523 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-15 02:53:50,523 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:53:50,523 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:50,523 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:53:50,523 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:53:50,523 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:53:50,523 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-15 02:53:50,523 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-15 02:53:50,523 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-15 02:53:50,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-15 02:53:50,524 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-15 02:53:50,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-15 02:53:50,623 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-15 02:53:50,624 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-15 02:53:50,625 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-15 02:53:50,626 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-15 02:53:50,626 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-15 02:53:50,626 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-15 02:53:50,626 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-15 02:53:50,626 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-15 02:53:50,626 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-15 02:53:50,626 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:53:50,626 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:53:50,627 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:50,627 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-15 02:53:50,627 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-15 02:53:50,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-15 02:53:50,627 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:50,627 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-15 02:53:50,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:53:50,628 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:53:50,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-15 02:53:50,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:53:50,628 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-15 02:53:50,628 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-15 02:53:50,628 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-15 02:53:50,628 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-15 02:53:50,628 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-15 02:53:50,628 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-15 02:53:50,630 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-15 02:53:50,630 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-15 02:53:50,630 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681527213260.c44270c3412c22e2c805f466aae6ff56. 2023-04-15 02:53:50,630 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681527213260.c44270c3412c22e2c805f466aae6ff56. started... 2023-04-15 02:53:50,630 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681527213260.c44270c3412c22e2c805f466aae6ff56. as already flushing 2023-04-15 02:53:50,630 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681527213260.c44270c3412c22e2c805f466aae6ff56. 2023-04-15 02:53:50,630 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-15 02:53:50,630 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-15 02:53:50,630 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:50,630 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-15 02:53:50,630 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34681,1681527209133' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-15 02:53:50,631 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:50,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:50,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:53:50,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:53:50,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:53:50,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:53:50,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:53:50,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:50,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:53:50,633 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:53:50,633 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34681,1681527209133' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-15 02:53:50,633 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:53:50,633 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-15 02:53:50,633 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-15 02:53:50,633 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@666cc47b[Count = 0] remaining members to acquire global barrier 2023-04-15 02:53:50,633 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:53:50,634 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:53:50,634 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:53:50,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:53:50,634 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-15 02:53:50,634 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-15 02:53:50,634 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34681,1681527209133' in zk 2023-04-15 02:53:50,634 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:50,634 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-15 02:53:50,634 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-15 02:53:50,634 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:50,635 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-15 02:53:50,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:50,635 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-15 02:53:50,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:53:50,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:53:50,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:53:50,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:53:50,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:53:50,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:50,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:53:50,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:53:50,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:50,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34681,1681527209133': 2023-04-15 02:53:50,636 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34681,1681527209133' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-15 02:53:50,636 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-15 02:53:50,636 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-15 02:53:50,636 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-15 02:53:50,637 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-15 02:53:50,637 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-15 02:53:50,639 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:53:50,639 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:53:50,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:53:50,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:53:50,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:53:50,639 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:53:50,639 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-15 02:53:50,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:53:50,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:53:50,639 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-15 02:53:50,640 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:50,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-15 02:53:50,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:53:50,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:53:50,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:53:50,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:53:50,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:50,641 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:50,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:53:50,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:53:50,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:50,643 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-15 02:53:50,643 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:50,643 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-15 02:53:50,643 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:53:50,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-15 02:53:50,643 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:53:50,643 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-15 02:53:50,643 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:50,643 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-15 02:53:50,643 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:53:50,643 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-15 02:53:50,643 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-15 02:53:50,643 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:53:50,643 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:53:50,643 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-15 02:53:50,643 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-15 02:53:50,643 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-15 02:53:50,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-15 02:53:50,657 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c44270c3412c22e2c805f466aae6ff56, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:53:50,658 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34681] ipc.CallRunner(144): callId: 141 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:38504 deadline: 1681527290657, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c44270c3412c22e2c805f466aae6ff56, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:50,687 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=414 (bloomFilter=true), to=hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/c44270c3412c22e2c805f466aae6ff56/.tmp/C/37ee1c600b8f4fe794a27f1f1edfa0fd 2023-04-15 02:53:50,691 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/c44270c3412c22e2c805f466aae6ff56/.tmp/A/9adad4a9679d4598acc5a03552f1531e as hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/c44270c3412c22e2c805f466aae6ff56/A/9adad4a9679d4598acc5a03552f1531e 2023-04-15 02:53:50,696 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/c44270c3412c22e2c805f466aae6ff56/A/9adad4a9679d4598acc5a03552f1531e, entries=150, sequenceid=414, filesize=11.9 K 2023-04-15 02:53:50,697 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/c44270c3412c22e2c805f466aae6ff56/.tmp/B/7cec6ef3d3de46b88188034a7cdb0313 as hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/c44270c3412c22e2c805f466aae6ff56/B/7cec6ef3d3de46b88188034a7cdb0313 2023-04-15 02:53:50,702 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/c44270c3412c22e2c805f466aae6ff56/B/7cec6ef3d3de46b88188034a7cdb0313, entries=150, sequenceid=414, filesize=11.9 K 2023-04-15 02:53:50,703 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/c44270c3412c22e2c805f466aae6ff56/.tmp/C/37ee1c600b8f4fe794a27f1f1edfa0fd as hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/c44270c3412c22e2c805f466aae6ff56/C/37ee1c600b8f4fe794a27f1f1edfa0fd 2023-04-15 02:53:50,708 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/c44270c3412c22e2c805f466aae6ff56/C/37ee1c600b8f4fe794a27f1f1edfa0fd, entries=150, sequenceid=414, filesize=11.9 K 2023-04-15 02:53:50,709 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=131.94 KB/135110 for c44270c3412c22e2c805f466aae6ff56 in 1262ms, sequenceid=414, compaction requested=true 2023-04-15 02:53:50,709 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for c44270c3412c22e2c805f466aae6ff56: 2023-04-15 02:53:50,709 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-15 02:53:50,709 DEBUG [RS:0;jenkins-hbase20:34681-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-15 02:53:50,709 DEBUG [RS:0;jenkins-hbase20:34681-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-15 02:53:50,709 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-15 02:53:50,709 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-15 02:53:50,710 DEBUG [RS:0;jenkins-hbase20:34681-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37308 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-15 02:53:50,710 DEBUG [RS:0;jenkins-hbase20:34681-shortCompactions-0] regionserver.HStore(1912): c44270c3412c22e2c805f466aae6ff56/A is initiating minor compaction (all files) 2023-04-15 02:53:50,710 INFO [RS:0;jenkins-hbase20:34681-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of c44270c3412c22e2c805f466aae6ff56/A in TestAcidGuarantees,,1681527213260.c44270c3412c22e2c805f466aae6ff56. 2023-04-15 02:53:50,710 INFO [RS:0;jenkins-hbase20:34681-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/c44270c3412c22e2c805f466aae6ff56/A/2573076332df46de97607613d26daddd, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/c44270c3412c22e2c805f466aae6ff56/A/3a38a999ff8441b7b08fcc09067aca06, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/c44270c3412c22e2c805f466aae6ff56/A/9adad4a9679d4598acc5a03552f1531e] into tmpdir=hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/c44270c3412c22e2c805f466aae6ff56/.tmp, totalSize=36.4 K 2023-04-15 02:53:50,711 DEBUG [RS:0;jenkins-hbase20:34681-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37308 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-15 02:53:50,711 DEBUG [RS:0;jenkins-hbase20:34681-longCompactions-0] regionserver.HStore(1912): c44270c3412c22e2c805f466aae6ff56/B is initiating minor compaction (all files) 2023-04-15 02:53:50,711 INFO [RS:0;jenkins-hbase20:34681-longCompactions-0] regionserver.HRegion(2259): Starting compaction of c44270c3412c22e2c805f466aae6ff56/B in TestAcidGuarantees,,1681527213260.c44270c3412c22e2c805f466aae6ff56. 2023-04-15 02:53:50,711 DEBUG [RS:0;jenkins-hbase20:34681-shortCompactions-0] compactions.Compactor(207): Compacting 2573076332df46de97607613d26daddd, keycount=150, bloomtype=ROW, size=12.6 K, encoding=NONE, compression=NONE, seqNum=390, earliestPutTs=1681527226572 2023-04-15 02:53:50,711 INFO [RS:0;jenkins-hbase20:34681-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/c44270c3412c22e2c805f466aae6ff56/B/fcd3cf354e5140479901c13ba09dae61, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/c44270c3412c22e2c805f466aae6ff56/B/9afaa7d2a9be47d185532e68809bfe92, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/c44270c3412c22e2c805f466aae6ff56/B/7cec6ef3d3de46b88188034a7cdb0313] into tmpdir=hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/c44270c3412c22e2c805f466aae6ff56/.tmp, totalSize=36.4 K 2023-04-15 02:53:50,711 DEBUG [RS:0;jenkins-hbase20:34681-shortCompactions-0] compactions.Compactor(207): Compacting 3a38a999ff8441b7b08fcc09067aca06, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=400, earliestPutTs=1681527227704 2023-04-15 02:53:50,711 DEBUG [RS:0;jenkins-hbase20:34681-longCompactions-0] compactions.Compactor(207): Compacting fcd3cf354e5140479901c13ba09dae61, keycount=150, bloomtype=ROW, size=12.6 K, encoding=NONE, compression=NONE, seqNum=390, earliestPutTs=1681527226572 2023-04-15 02:53:50,712 DEBUG [RS:0;jenkins-hbase20:34681-shortCompactions-0] compactions.Compactor(207): Compacting 9adad4a9679d4598acc5a03552f1531e, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=414, earliestPutTs=1681527229430 2023-04-15 02:53:50,712 DEBUG [RS:0;jenkins-hbase20:34681-longCompactions-0] compactions.Compactor(207): Compacting 9afaa7d2a9be47d185532e68809bfe92, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=400, earliestPutTs=1681527227704 2023-04-15 02:53:50,712 DEBUG [RS:0;jenkins-hbase20:34681-longCompactions-0] compactions.Compactor(207): Compacting 7cec6ef3d3de46b88188034a7cdb0313, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=414, earliestPutTs=1681527229430 2023-04-15 02:53:50,721 INFO [RS:0;jenkins-hbase20:34681-longCompactions-0] throttle.PressureAwareThroughputController(145): c44270c3412c22e2c805f466aae6ff56#B#compaction#75 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-15 02:53:50,727 INFO [RS:0;jenkins-hbase20:34681-shortCompactions-0] throttle.PressureAwareThroughputController(145): c44270c3412c22e2c805f466aae6ff56#A#compaction#76 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-15 02:53:50,744 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-15 02:53:50,744 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-15 02:53:50,746 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-15 02:53:50,747 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-15 02:53:50,747 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-15 02:53:50,747 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-15 02:53:50,747 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-15 02:53:50,747 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-15 02:53:50,747 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-15 02:53:50,748 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:53:50,748 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:53:50,749 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-15 02:53:50,749 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-15 02:53:50,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-15 02:53:50,749 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:50,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:53:50,749 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:50,749 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-15 02:53:50,750 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:53:50,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-15 02:53:50,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:53:50,750 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-15 02:53:50,752 DEBUG [RS:0;jenkins-hbase20:34681-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/c44270c3412c22e2c805f466aae6ff56/.tmp/B/00a3a9b0abc448ae9c1f19b38c7583ff as hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/c44270c3412c22e2c805f466aae6ff56/B/00a3a9b0abc448ae9c1f19b38c7583ff 2023-04-15 02:53:50,754 DEBUG [RS:0;jenkins-hbase20:34681-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/c44270c3412c22e2c805f466aae6ff56/.tmp/A/58061441bf484cf5a57757d389a55706 as hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/c44270c3412c22e2c805f466aae6ff56/A/58061441bf484cf5a57757d389a55706 2023-04-15 02:53:50,754 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-15 02:53:50,754 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-15 02:53:50,754 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-15 02:53:50,754 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-15 02:53:50,754 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-15 02:53:50,765 INFO [RS:0;jenkins-hbase20:34681-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in c44270c3412c22e2c805f466aae6ff56/B of c44270c3412c22e2c805f466aae6ff56 into 00a3a9b0abc448ae9c1f19b38c7583ff(size=12.7 K), total size for store is 12.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-15 02:53:50,765 INFO [RS:0;jenkins-hbase20:34681-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in c44270c3412c22e2c805f466aae6ff56/A of c44270c3412c22e2c805f466aae6ff56 into 58061441bf484cf5a57757d389a55706(size=12.7 K), total size for store is 12.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-15 02:53:50,766 DEBUG [RS:0;jenkins-hbase20:34681-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for c44270c3412c22e2c805f466aae6ff56: 2023-04-15 02:53:50,766 DEBUG [RS:0;jenkins-hbase20:34681-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for c44270c3412c22e2c805f466aae6ff56: 2023-04-15 02:53:50,766 INFO [RS:0;jenkins-hbase20:34681-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681527213260.c44270c3412c22e2c805f466aae6ff56., storeName=c44270c3412c22e2c805f466aae6ff56/B, priority=13, startTime=1681527230709; duration=0sec 2023-04-15 02:53:50,766 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-15 02:53:50,766 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-15 02:53:50,766 DEBUG [RS:0;jenkins-hbase20:34681-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-15 02:53:50,766 DEBUG [RS:0;jenkins-hbase20:34681-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-15 02:53:50,766 INFO [RS:0;jenkins-hbase20:34681-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681527213260.c44270c3412c22e2c805f466aae6ff56., storeName=c44270c3412c22e2c805f466aae6ff56/A, priority=13, startTime=1681527230709; duration=0sec 2023-04-15 02:53:50,766 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681527213260.c44270c3412c22e2c805f466aae6ff56. 2023-04-15 02:53:50,766 DEBUG [RS:0;jenkins-hbase20:34681-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-15 02:53:50,766 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681527213260.c44270c3412c22e2c805f466aae6ff56. started... 2023-04-15 02:53:50,766 INFO [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing c44270c3412c22e2c805f466aae6ff56 3/3 column families, dataSize=131.94 KB heapSize=383.31 KB 2023-04-15 02:53:50,767 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c44270c3412c22e2c805f466aae6ff56, store=A 2023-04-15 02:53:50,767 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-15 02:53:50,767 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c44270c3412c22e2c805f466aae6ff56, store=B 2023-04-15 02:53:50,767 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-15 02:53:50,767 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c44270c3412c22e2c805f466aae6ff56, store=C 2023-04-15 02:53:50,767 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-15 02:53:50,767 DEBUG [RS:0;jenkins-hbase20:34681-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37308 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-15 02:53:50,767 DEBUG [RS:0;jenkins-hbase20:34681-longCompactions-0] regionserver.HStore(1912): c44270c3412c22e2c805f466aae6ff56/C is initiating minor compaction (all files) 2023-04-15 02:53:50,767 INFO [RS:0;jenkins-hbase20:34681-longCompactions-0] regionserver.HRegion(2259): Starting compaction of c44270c3412c22e2c805f466aae6ff56/C in TestAcidGuarantees,,1681527213260.c44270c3412c22e2c805f466aae6ff56. 2023-04-15 02:53:50,767 INFO [RS:0;jenkins-hbase20:34681-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/c44270c3412c22e2c805f466aae6ff56/C/3de6cdf4cc5f47ff9def540e573bb52d, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/c44270c3412c22e2c805f466aae6ff56/C/f6f90e98a27f4485ba53d503154f7322, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/c44270c3412c22e2c805f466aae6ff56/C/37ee1c600b8f4fe794a27f1f1edfa0fd] into tmpdir=hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/c44270c3412c22e2c805f466aae6ff56/.tmp, totalSize=36.4 K 2023-04-15 02:53:50,768 DEBUG [RS:0;jenkins-hbase20:34681-longCompactions-0] compactions.Compactor(207): Compacting 3de6cdf4cc5f47ff9def540e573bb52d, keycount=150, bloomtype=ROW, size=12.6 K, encoding=NONE, compression=NONE, seqNum=390, earliestPutTs=1681527226572 2023-04-15 02:53:50,768 DEBUG [RS:0;jenkins-hbase20:34681-longCompactions-0] compactions.Compactor(207): Compacting f6f90e98a27f4485ba53d503154f7322, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=400, earliestPutTs=1681527227704 2023-04-15 02:53:50,768 DEBUG [RS:0;jenkins-hbase20:34681-longCompactions-0] compactions.Compactor(207): Compacting 37ee1c600b8f4fe794a27f1f1edfa0fd, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=414, earliestPutTs=1681527229430 2023-04-15 02:53:50,779 INFO [RS:0;jenkins-hbase20:34681-longCompactions-0] throttle.PressureAwareThroughputController(145): c44270c3412c22e2c805f466aae6ff56#C#compaction#78 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-15 02:53:50,781 INFO [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=448 (bloomFilter=true), to=hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/c44270c3412c22e2c805f466aae6ff56/.tmp/A/f5429821fb914e68ba7a97b8cf6ad6ad 2023-04-15 02:53:51,190 DEBUG [RS:0;jenkins-hbase20:34681-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/c44270c3412c22e2c805f466aae6ff56/.tmp/C/4f1c63d6ddf542c6bd87f7fc14334ff0 as hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/c44270c3412c22e2c805f466aae6ff56/C/4f1c63d6ddf542c6bd87f7fc14334ff0 2023-04-15 02:53:51,194 INFO [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=448 (bloomFilter=true), to=hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/c44270c3412c22e2c805f466aae6ff56/.tmp/B/4eb9fadc993646b894e80997d6e745ca 2023-04-15 02:53:51,195 INFO [RS:0;jenkins-hbase20:34681-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in c44270c3412c22e2c805f466aae6ff56/C of c44270c3412c22e2c805f466aae6ff56 into 4f1c63d6ddf542c6bd87f7fc14334ff0(size=12.7 K), total size for store is 12.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-15 02:53:51,195 DEBUG [RS:0;jenkins-hbase20:34681-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for c44270c3412c22e2c805f466aae6ff56: 2023-04-15 02:53:51,195 INFO [RS:0;jenkins-hbase20:34681-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681527213260.c44270c3412c22e2c805f466aae6ff56., storeName=c44270c3412c22e2c805f466aae6ff56/C, priority=13, startTime=1681527230709; duration=0sec 2023-04-15 02:53:51,196 DEBUG [RS:0;jenkins-hbase20:34681-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-15 02:53:51,608 INFO [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=64.85 KB at sequenceid=448 (bloomFilter=true), to=hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/c44270c3412c22e2c805f466aae6ff56/.tmp/C/81b3e7d008db4b72a7cea39386d1e7a7 2023-04-15 02:53:51,612 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/c44270c3412c22e2c805f466aae6ff56/.tmp/A/f5429821fb914e68ba7a97b8cf6ad6ad as hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/c44270c3412c22e2c805f466aae6ff56/A/f5429821fb914e68ba7a97b8cf6ad6ad 2023-04-15 02:53:51,618 INFO [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/c44270c3412c22e2c805f466aae6ff56/A/f5429821fb914e68ba7a97b8cf6ad6ad, entries=150, sequenceid=448, filesize=11.9 K 2023-04-15 02:53:51,619 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/c44270c3412c22e2c805f466aae6ff56/.tmp/B/4eb9fadc993646b894e80997d6e745ca as hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/c44270c3412c22e2c805f466aae6ff56/B/4eb9fadc993646b894e80997d6e745ca 2023-04-15 02:53:51,624 INFO [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/c44270c3412c22e2c805f466aae6ff56/B/4eb9fadc993646b894e80997d6e745ca, entries=150, sequenceid=448, filesize=11.9 K 2023-04-15 02:53:51,624 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/c44270c3412c22e2c805f466aae6ff56/.tmp/C/81b3e7d008db4b72a7cea39386d1e7a7 as hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/c44270c3412c22e2c805f466aae6ff56/C/81b3e7d008db4b72a7cea39386d1e7a7 2023-04-15 02:53:51,629 INFO [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/c44270c3412c22e2c805f466aae6ff56/C/81b3e7d008db4b72a7cea39386d1e7a7, entries=150, sequenceid=448, filesize=11.9 K 2023-04-15 02:53:51,630 INFO [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~131.94 KB/135110, heapSize ~383.27 KB/392464, currentSize=0 B/0 for c44270c3412c22e2c805f466aae6ff56 in 864ms, sequenceid=448, compaction requested=false 2023-04-15 02:53:51,630 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for c44270c3412c22e2c805f466aae6ff56: 2023-04-15 02:53:51,630 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681527213260.c44270c3412c22e2c805f466aae6ff56. 2023-04-15 02:53:51,630 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-15 02:53:51,630 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-15 02:53:51,630 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:51,630 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-15 02:53:51,630 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34681,1681527209133' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-15 02:53:51,631 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:51,631 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:53:51,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:51,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:53:51,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:53:51,631 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:53:51,631 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-15 02:53:51,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:53:51,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:53:51,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:53:51,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:51,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:53:51,632 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34681,1681527209133' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-15 02:53:51,632 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@15912c42[Count = 0] remaining members to acquire global barrier 2023-04-15 02:53:51,632 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-15 02:53:51,633 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:53:51,633 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:53:51,633 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:53:51,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:53:51,633 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-15 02:53:51,633 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-15 02:53:51,633 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:51,633 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-15 02:53:51,633 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34681,1681527209133' in zk 2023-04-15 02:53:51,634 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-15 02:53:51,634 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:51,634 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-15 02:53:51,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:51,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:53:51,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:53:51,635 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-15 02:53:51,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:53:51,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:53:51,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:53:51,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:51,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:53:51,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:53:51,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:51,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34681,1681527209133': 2023-04-15 02:53:51,637 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34681,1681527209133' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-15 02:53:51,637 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-15 02:53:51,637 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-15 02:53:51,637 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-15 02:53:51,637 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-15 02:53:51,637 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-15 02:53:51,638 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:53:51,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:53:51,643 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:53:51,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:53:51,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:53:51,643 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:53:51,643 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-15 02:53:51,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:53:51,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:53:51,643 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-15 02:53:51,643 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:51,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-15 02:53:51,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:53:51,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:53:51,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:53:51,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:53:51,646 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:51,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:51,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:53:51,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:53:51,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:51,652 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-15 02:53:51,652 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:51,652 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-15 02:53:51,652 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-15 02:53:51,652 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:53:51,652 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-15 02:53:51,652 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-15 02:53:51,652 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-15 02:53:51,652 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-15 02:53:51,652 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-15 02:53:51,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-15 02:53:51,652 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:53:51,653 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-15 02:53:51,653 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:51,653 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:53:51,653 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:53:51,653 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:53:51,653 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-15 02:53:51,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-15 02:53:51,673 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34681] regionserver.HRegion(9158): Flush requested on c44270c3412c22e2c805f466aae6ff56 2023-04-15 02:53:51,674 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing c44270c3412c22e2c805f466aae6ff56 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-15 02:53:51,674 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c44270c3412c22e2c805f466aae6ff56, store=A 2023-04-15 02:53:51,674 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-15 02:53:51,674 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c44270c3412c22e2c805f466aae6ff56, store=B 2023-04-15 02:53:51,674 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-15 02:53:51,674 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c44270c3412c22e2c805f466aae6ff56, store=C 2023-04-15 02:53:51,674 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-15 02:53:51,699 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=459 (bloomFilter=true), to=hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/c44270c3412c22e2c805f466aae6ff56/.tmp/A/3e42386781194a9bab119952e28e6bba 2023-04-15 02:53:51,725 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1681527231698, max timestamp=1681527231722Num uniques 859; 2023-04-15 02:53:51,725 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1681527231698, max timestamp=1681527231722Num uniques 859; 2023-04-15 02:53:51,725 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1681527231698, max timestamp=1681527231722Num uniques 859; 2023-04-15 02:53:51,725 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-4] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-04-15 02:53:51,725 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-9] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-04-15 02:53:51,725 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-8] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-04-15 02:53:51,753 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-15 02:53:51,753 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-15 02:53:51,755 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-15 02:53:51,756 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-15 02:53:51,756 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-15 02:53:51,756 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-15 02:53:51,756 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-15 02:53:51,756 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-15 02:53:51,756 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-15 02:53:51,756 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:53:51,756 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:53:51,757 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-15 02:53:51,757 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-15 02:53:51,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-15 02:53:51,757 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:51,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:53:51,757 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:51,757 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-15 02:53:51,757 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:53:51,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-15 02:53:51,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:53:51,757 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-15 02:53:51,758 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-15 02:53:51,758 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-15 02:53:51,758 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-15 02:53:51,758 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-15 02:53:51,758 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-15 02:53:51,758 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-15 02:53:51,758 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-15 02:53:51,758 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681527213260.c44270c3412c22e2c805f466aae6ff56. 2023-04-15 02:53:51,758 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681527213260.c44270c3412c22e2c805f466aae6ff56. started... 2023-04-15 02:53:51,758 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681527213260.c44270c3412c22e2c805f466aae6ff56. as already flushing 2023-04-15 02:53:51,758 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681527213260.c44270c3412c22e2c805f466aae6ff56. 2023-04-15 02:53:51,758 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-15 02:53:51,758 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-15 02:53:51,758 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:51,758 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-15 02:53:51,758 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34681,1681527209133' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-15 02:53:51,759 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:53:51,759 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:51,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:51,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:53:51,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:53:51,759 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:53:51,759 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-15 02:53:51,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:53:51,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:53:51,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:53:51,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:51,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:53:51,762 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34681,1681527209133' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-15 02:53:51,762 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3909af3d[Count = 0] remaining members to acquire global barrier 2023-04-15 02:53:51,762 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-15 02:53:51,762 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:53:51,762 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:53:51,762 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:53:51,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:53:51,763 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-15 02:53:51,763 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-15 02:53:51,763 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34681,1681527209133' in zk 2023-04-15 02:53:51,763 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:51,763 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-15 02:53:51,763 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:51,764 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-15 02:53:51,764 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-15 02:53:51,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:51,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:53:51,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:53:51,764 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-15 02:53:51,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:53:51,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:53:51,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:53:51,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:51,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:53:51,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:53:51,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:51,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34681,1681527209133': 2023-04-15 02:53:51,765 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34681,1681527209133' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-15 02:53:51,765 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-15 02:53:51,766 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-15 02:53:51,766 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-15 02:53:51,766 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-15 02:53:51,766 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-15 02:53:51,767 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:53:51,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:53:51,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:53:51,767 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:53:51,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:53:51,767 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:53:51,767 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-15 02:53:51,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:53:51,767 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-15 02:53:51,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-15 02:53:51,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:53:51,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:53:51,768 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:51,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:53:51,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:53:51,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:53:51,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:51,769 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:51,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:53:51,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:53:51,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:51,772 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c44270c3412c22e2c805f466aae6ff56, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:53:51,772 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34681] ipc.CallRunner(144): callId: 182 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:38504 deadline: 1681527291769, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c44270c3412c22e2c805f466aae6ff56, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:51,773 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:51,773 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:53:51,773 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-15 02:53:51,773 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-15 02:53:51,773 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-15 02:53:51,773 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-15 02:53:51,773 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-15 02:53:51,773 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-15 02:53:51,774 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-15 02:53:51,773 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-15 02:53:51,773 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:53:51,774 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-15 02:53:51,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-15 02:53:51,774 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:51,774 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:53:51,774 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:53:51,774 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:53:51,774 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-15 02:53:51,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-15 02:53:51,845 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c44270c3412c22e2c805f466aae6ff56, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:53:51,845 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34681] ipc.CallRunner(144): callId: 144 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:38542 deadline: 1681527291843, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c44270c3412c22e2c805f466aae6ff56, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:51,846 DEBUG [Thread-149] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4139 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c44270c3412c22e2c805f466aae6ff56, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_2' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1681527213260.c44270c3412c22e2c805f466aae6ff56., hostname=jenkins-hbase20.apache.org,34681,1681527209133, seqNum=2, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c44270c3412c22e2c805f466aae6ff56, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor31.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c44270c3412c22e2c805f466aae6ff56, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-15 02:53:51,855 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c44270c3412c22e2c805f466aae6ff56, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:53:51,855 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34681] ipc.CallRunner(144): callId: 149 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:38520 deadline: 1681527291853, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c44270c3412c22e2c805f466aae6ff56, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:51,857 DEBUG [Thread-145] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4146 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c44270c3412c22e2c805f466aae6ff56, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_2' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1681527213260.c44270c3412c22e2c805f466aae6ff56., hostname=jenkins-hbase20.apache.org,34681,1681527209133, seqNum=2, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c44270c3412c22e2c805f466aae6ff56, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor31.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c44270c3412c22e2c805f466aae6ff56, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-15 02:53:51,873 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c44270c3412c22e2c805f466aae6ff56, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:53:51,873 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34681] ipc.CallRunner(144): callId: 184 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:38504 deadline: 1681527291873, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c44270c3412c22e2c805f466aae6ff56, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:51,874 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-15 02:53:51,874 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-15 02:53:51,876 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-15 02:53:51,876 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-15 02:53:51,876 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-15 02:53:51,876 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-15 02:53:51,877 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-15 02:53:51,877 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-15 02:53:51,877 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-15 02:53:51,877 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:53:51,877 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:53:51,878 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-15 02:53:51,878 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-15 02:53:51,878 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:51,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-15 02:53:51,878 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:51,878 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-15 02:53:51,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:53:51,878 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:53:51,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-15 02:53:51,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:53:51,879 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-15 02:53:51,879 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-15 02:53:51,879 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-15 02:53:51,879 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-15 02:53:51,880 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-15 02:53:51,880 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-15 02:53:51,880 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-15 02:53:51,880 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-15 02:53:51,880 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681527213260.c44270c3412c22e2c805f466aae6ff56. 2023-04-15 02:53:51,880 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681527213260.c44270c3412c22e2c805f466aae6ff56. started... 2023-04-15 02:53:51,880 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681527213260.c44270c3412c22e2c805f466aae6ff56. as already flushing 2023-04-15 02:53:51,880 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681527213260.c44270c3412c22e2c805f466aae6ff56. 2023-04-15 02:53:51,880 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-15 02:53:51,880 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-15 02:53:51,880 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:51,880 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-15 02:53:51,880 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34681,1681527209133' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-15 02:53:51,881 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:51,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:51,881 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:53:51,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:53:51,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:53:51,881 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:53:51,881 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-15 02:53:51,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:53:51,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:53:51,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:53:51,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:51,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:53:51,883 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34681,1681527209133' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-15 02:53:51,883 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@516a6091[Count = 0] remaining members to acquire global barrier 2023-04-15 02:53:51,883 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-15 02:53:51,883 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:53:51,883 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:53:51,883 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:53:51,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:53:51,883 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-15 02:53:51,883 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-15 02:53:51,883 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:51,883 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-15 02:53:51,883 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34681,1681527209133' in zk 2023-04-15 02:53:51,884 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:51,884 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-15 02:53:51,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:51,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:53:51,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:53:51,884 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-15 02:53:51,884 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-15 02:53:51,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:53:51,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:53:51,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:53:51,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:51,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:53:51,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:53:51,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:51,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34681,1681527209133': 2023-04-15 02:53:51,886 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34681,1681527209133' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-15 02:53:51,886 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-15 02:53:51,886 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-15 02:53:51,886 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-15 02:53:51,886 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-15 02:53:51,886 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-15 02:53:51,887 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:53:51,887 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:53:51,887 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-15 02:53:51,887 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:53:51,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:53:51,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:53:51,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:53:51,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:53:51,887 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-15 02:53:51,887 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:51,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-15 02:53:51,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:53:51,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:53:51,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:53:51,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:53:51,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:53:51,888 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:51,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:51,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:53:51,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:53:51,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:51,890 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:51,890 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-15 02:53:51,890 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:53:51,890 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-15 02:53:51,890 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:53:51,890 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-15 02:53:51,890 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-15 02:53:51,890 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-15 02:53:51,890 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-15 02:53:51,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-15 02:53:51,890 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-15 02:53:51,890 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-15 02:53:51,890 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:51,890 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-15 02:53:51,890 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:53:51,890 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-15 02:53:51,890 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:53:51,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-15 02:53:51,890 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:53:51,990 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-15 02:53:51,991 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-15 02:53:51,992 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-15 02:53:51,993 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-15 02:53:51,993 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-15 02:53:51,993 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-15 02:53:51,993 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-15 02:53:51,993 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-15 02:53:51,993 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-15 02:53:51,993 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:53:51,993 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:53:51,994 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-15 02:53:51,994 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:51,994 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-15 02:53:51,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-15 02:53:51,994 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:51,994 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-15 02:53:51,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:53:51,995 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:53:51,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-15 02:53:51,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:53:51,995 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-15 02:53:51,995 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-15 02:53:51,995 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-15 02:53:51,995 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-15 02:53:51,995 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-15 02:53:51,995 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-15 02:53:51,999 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-15 02:53:51,999 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-15 02:53:51,999 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681527213260.c44270c3412c22e2c805f466aae6ff56. 2023-04-15 02:53:51,999 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681527213260.c44270c3412c22e2c805f466aae6ff56. started... 2023-04-15 02:53:51,999 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681527213260.c44270c3412c22e2c805f466aae6ff56. as already flushing 2023-04-15 02:53:51,999 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681527213260.c44270c3412c22e2c805f466aae6ff56. 2023-04-15 02:53:51,999 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-15 02:53:51,999 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-15 02:53:51,999 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:51,999 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-15 02:53:51,999 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34681,1681527209133' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-15 02:53:52,000 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:53:52,000 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:52,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:52,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:53:52,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:53:52,001 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:53:52,001 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-15 02:53:52,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:53:52,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:53:52,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:53:52,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:52,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:53:52,002 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34681,1681527209133' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-15 02:53:52,002 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@512529db[Count = 0] remaining members to acquire global barrier 2023-04-15 02:53:52,002 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-15 02:53:52,002 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:53:52,002 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:53:52,002 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:53:52,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:53:52,003 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-15 02:53:52,003 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-15 02:53:52,003 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34681,1681527209133' in zk 2023-04-15 02:53:52,003 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:52,003 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-15 02:53:52,003 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:52,003 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-15 02:53:52,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:52,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:53:52,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:53:52,003 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-15 02:53:52,003 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-15 02:53:52,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:53:52,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:53:52,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:53:52,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:52,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:53:52,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:53:52,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:52,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34681,1681527209133': 2023-04-15 02:53:52,005 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34681,1681527209133' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-15 02:53:52,005 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-15 02:53:52,005 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-15 02:53:52,005 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-15 02:53:52,005 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-15 02:53:52,005 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-15 02:53:52,006 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:53:52,006 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:53:52,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:53:52,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:53:52,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:53:52,006 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:53:52,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:53:52,006 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-15 02:53:52,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:53:52,006 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-15 02:53:52,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-15 02:53:52,006 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:52,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:53:52,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:53:52,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:53:52,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:53:52,007 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:52,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:52,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:53:52,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:53:52,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:52,009 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:52,009 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:53:52,009 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:53:52,009 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:52,009 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-15 02:53:52,009 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-15 02:53:52,009 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-15 02:53:52,009 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-15 02:53:52,009 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:53:52,009 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-15 02:53:52,009 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-15 02:53:52,009 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-15 02:53:52,009 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:53:52,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-15 02:53:52,009 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-15 02:53:52,009 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:53:52,009 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-15 02:53:52,009 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-15 02:53:52,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-15 02:53:52,078 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c44270c3412c22e2c805f466aae6ff56, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:53:52,078 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34681] ipc.CallRunner(144): callId: 186 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:38504 deadline: 1681527292075, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c44270c3412c22e2c805f466aae6ff56, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:52,109 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-15 02:53:52,110 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-15 02:53:52,111 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-15 02:53:52,112 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-15 02:53:52,112 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-15 02:53:52,112 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-15 02:53:52,112 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-15 02:53:52,112 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-15 02:53:52,112 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-15 02:53:52,113 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:53:52,113 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:53:52,113 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:52,113 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-15 02:53:52,113 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-15 02:53:52,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-15 02:53:52,113 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:52,114 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-15 02:53:52,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:53:52,114 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:53:52,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-15 02:53:52,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:53:52,114 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-15 02:53:52,115 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-15 02:53:52,115 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-15 02:53:52,115 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=459 (bloomFilter=true), to=hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/c44270c3412c22e2c805f466aae6ff56/.tmp/B/0c1862f1233640a5810d6d90f85e4696 2023-04-15 02:53:52,115 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-15 02:53:52,115 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-15 02:53:52,115 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-15 02:53:52,115 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-15 02:53:52,115 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-15 02:53:52,115 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681527213260.c44270c3412c22e2c805f466aae6ff56. 2023-04-15 02:53:52,115 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681527213260.c44270c3412c22e2c805f466aae6ff56. started... 2023-04-15 02:53:52,115 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681527213260.c44270c3412c22e2c805f466aae6ff56. as already flushing 2023-04-15 02:53:52,116 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681527213260.c44270c3412c22e2c805f466aae6ff56. 2023-04-15 02:53:52,116 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-15 02:53:52,116 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-15 02:53:52,116 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:52,116 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-15 02:53:52,116 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34681,1681527209133' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-15 02:53:52,117 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:52,117 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:53:52,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:52,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:53:52,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:53:52,118 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:53:52,118 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-15 02:53:52,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:53:52,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:53:52,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:53:52,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:52,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:53:52,119 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34681,1681527209133' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-15 02:53:52,119 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4dcafc81[Count = 0] remaining members to acquire global barrier 2023-04-15 02:53:52,119 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-15 02:53:52,119 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:53:52,120 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:53:52,120 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:53:52,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:53:52,120 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-15 02:53:52,120 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-15 02:53:52,120 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34681,1681527209133' in zk 2023-04-15 02:53:52,120 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:52,120 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-15 02:53:52,121 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-15 02:53:52,121 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:52,121 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-15 02:53:52,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:52,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:53:52,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:53:52,121 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-15 02:53:52,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:53:52,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:53:52,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:53:52,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:52,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:53:52,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:53:52,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:52,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34681,1681527209133': 2023-04-15 02:53:52,124 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34681,1681527209133' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-15 02:53:52,124 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-15 02:53:52,124 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-15 02:53:52,124 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-15 02:53:52,124 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-15 02:53:52,124 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-15 02:53:52,125 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:53:52,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:53:52,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:53:52,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:53:52,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:53:52,126 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:52,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:53:52,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:53:52,127 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:53:52,127 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-15 02:53:52,127 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:53:52,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:53:52,127 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-15 02:53:52,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-15 02:53:52,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:53:52,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:53:52,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:52,128 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:52,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:53:52,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:53:52,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:52,129 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:52,129 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:53:52,129 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-15 02:53:52,130 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-15 02:53:52,130 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-15 02:53:52,129 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:53:52,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-15 02:53:52,130 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-15 02:53:52,130 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-15 02:53:52,130 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-15 02:53:52,130 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-15 02:53:52,130 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:52,130 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-15 02:53:52,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-15 02:53:52,130 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-15 02:53:52,130 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:53:52,130 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-15 02:53:52,130 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:53:52,130 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:53:52,230 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-15 02:53:52,231 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-15 02:53:52,233 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-15 02:53:52,234 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-15 02:53:52,234 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-15 02:53:52,234 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-15 02:53:52,234 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-15 02:53:52,234 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-15 02:53:52,234 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-15 02:53:52,235 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:53:52,235 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:53:52,235 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-15 02:53:52,236 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-15 02:53:52,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-15 02:53:52,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:53:52,236 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:52,236 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:53:52,237 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:52,237 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-15 02:53:52,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-15 02:53:52,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:53:52,237 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-15 02:53:52,237 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-15 02:53:52,237 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-15 02:53:52,237 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-15 02:53:52,237 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-15 02:53:52,242 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-15 02:53:52,242 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-15 02:53:52,242 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681527213260.c44270c3412c22e2c805f466aae6ff56. 2023-04-15 02:53:52,242 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681527213260.c44270c3412c22e2c805f466aae6ff56. started... 2023-04-15 02:53:52,242 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681527213260.c44270c3412c22e2c805f466aae6ff56. as already flushing 2023-04-15 02:53:52,242 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681527213260.c44270c3412c22e2c805f466aae6ff56. 2023-04-15 02:53:52,242 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-15 02:53:52,242 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-15 02:53:52,242 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:52,242 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-15 02:53:52,242 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34681,1681527209133' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-15 02:53:52,252 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:52,252 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:53:52,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:52,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:53:52,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:53:52,253 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:53:52,253 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-15 02:53:52,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:53:52,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:53:52,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:53:52,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:52,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:53:52,254 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34681,1681527209133' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-15 02:53:52,254 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@39ced242[Count = 0] remaining members to acquire global barrier 2023-04-15 02:53:52,254 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-15 02:53:52,254 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:53:52,257 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:53:52,257 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:53:52,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:53:52,257 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-15 02:53:52,257 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-15 02:53:52,257 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34681,1681527209133' in zk 2023-04-15 02:53:52,258 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:52,258 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-15 02:53:52,258 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:52,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:52,258 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-15 02:53:52,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:53:52,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:53:52,258 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-15 02:53:52,259 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-15 02:53:52,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:53:52,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:53:52,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:53:52,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:52,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:53:52,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:53:52,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:52,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34681,1681527209133': 2023-04-15 02:53:52,261 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34681,1681527209133' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-15 02:53:52,261 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-15 02:53:52,261 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-15 02:53:52,261 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-15 02:53:52,261 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-15 02:53:52,261 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-15 02:53:52,263 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:53:52,263 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-15 02:53:52,263 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:53:52,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:53:52,263 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-15 02:53:52,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-15 02:53:52,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:53:52,266 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:53:52,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:53:52,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:53:52,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:53:52,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:53:52,266 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:52,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:53:52,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:53:52,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:53:52,268 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:52,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:52,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:53:52,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:53:52,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:52,270 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-15 02:53:52,270 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-15 02:53:52,270 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-15 02:53:52,270 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-15 02:53:52,270 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-15 02:53:52,270 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-15 02:53:52,270 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:52,270 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-15 02:53:52,270 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:53:52,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-15 02:53:52,270 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-15 02:53:52,270 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-15 02:53:52,270 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:53:52,270 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:52,270 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:53:52,270 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:53:52,270 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:53:52,270 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-15 02:53:52,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-15 02:53:52,370 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-15 02:53:52,371 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-15 02:53:52,372 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-15 02:53:52,373 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-15 02:53:52,373 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-15 02:53:52,373 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-15 02:53:52,373 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-15 02:53:52,373 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-15 02:53:52,373 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-15 02:53:52,373 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:53:52,373 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:53:52,375 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:52,375 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-15 02:53:52,375 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-15 02:53:52,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-15 02:53:52,375 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:52,375 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-15 02:53:52,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:53:52,376 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:53:52,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-15 02:53:52,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:53:52,376 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-15 02:53:52,376 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-15 02:53:52,376 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-15 02:53:52,376 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-15 02:53:52,376 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-15 02:53:52,376 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-15 02:53:52,376 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-15 02:53:52,376 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-15 02:53:52,376 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681527213260.c44270c3412c22e2c805f466aae6ff56. 2023-04-15 02:53:52,376 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681527213260.c44270c3412c22e2c805f466aae6ff56. started... 2023-04-15 02:53:52,376 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681527213260.c44270c3412c22e2c805f466aae6ff56. as already flushing 2023-04-15 02:53:52,376 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681527213260.c44270c3412c22e2c805f466aae6ff56. 2023-04-15 02:53:52,377 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-15 02:53:52,377 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-15 02:53:52,377 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:52,377 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-15 02:53:52,377 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34681,1681527209133' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-15 02:53:52,377 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:52,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:52,377 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:53:52,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:53:52,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:53:52,378 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:53:52,378 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-15 02:53:52,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:53:52,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:53:52,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:53:52,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:52,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:53:52,379 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34681,1681527209133' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-15 02:53:52,379 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-15 02:53:52,379 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3b224302[Count = 0] remaining members to acquire global barrier 2023-04-15 02:53:52,379 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:53:52,379 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:53:52,379 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:53:52,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:53:52,380 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-15 02:53:52,380 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-15 02:53:52,380 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:52,380 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-15 02:53:52,380 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34681,1681527209133' in zk 2023-04-15 02:53:52,381 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:52,381 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-15 02:53:52,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:52,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:53:52,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:53:52,381 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-15 02:53:52,381 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-15 02:53:52,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:53:52,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:53:52,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:53:52,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:52,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:53:52,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:53:52,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:52,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34681,1681527209133': 2023-04-15 02:53:52,383 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34681,1681527209133' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-15 02:53:52,383 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-15 02:53:52,383 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-15 02:53:52,383 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-15 02:53:52,383 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-15 02:53:52,383 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-15 02:53:52,384 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:53:52,384 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:53:52,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:53:52,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:53:52,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:53:52,384 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-15 02:53:52,384 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:53:52,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:53:52,384 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c44270c3412c22e2c805f466aae6ff56, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:53:52,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:53:52,384 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34681] ipc.CallRunner(144): callId: 188 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:38504 deadline: 1681527292381, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c44270c3412c22e2c805f466aae6ff56, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:52,384 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:52,384 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-15 02:53:52,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-15 02:53:52,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:53:52,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:53:52,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:53:52,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:53:52,385 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:52,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:52,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:53:52,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:53:52,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:52,386 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:52,386 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-15 02:53:52,386 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:53:52,386 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-15 02:53:52,386 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-15 02:53:52,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-15 02:53:52,387 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-15 02:53:52,387 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-15 02:53:52,387 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-15 02:53:52,387 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-15 02:53:52,386 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:53:52,387 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-15 02:53:52,387 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-15 02:53:52,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-15 02:53:52,387 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-15 02:53:52,387 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:52,387 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:53:52,387 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:53:52,387 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:53:52,487 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-15 02:53:52,488 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-15 02:53:52,489 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-15 02:53:52,490 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-15 02:53:52,490 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-15 02:53:52,490 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-15 02:53:52,490 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-15 02:53:52,490 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-15 02:53:52,490 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-15 02:53:52,490 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:53:52,490 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:53:52,491 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-15 02:53:52,491 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:52,491 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-15 02:53:52,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-15 02:53:52,491 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:52,491 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-15 02:53:52,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:53:52,492 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:53:52,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-15 02:53:52,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:53:52,492 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-15 02:53:52,492 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-15 02:53:52,492 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-15 02:53:52,492 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-15 02:53:52,492 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-15 02:53:52,492 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-15 02:53:52,492 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-15 02:53:52,493 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-15 02:53:52,493 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681527213260.c44270c3412c22e2c805f466aae6ff56. 2023-04-15 02:53:52,493 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681527213260.c44270c3412c22e2c805f466aae6ff56. started... 2023-04-15 02:53:52,493 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681527213260.c44270c3412c22e2c805f466aae6ff56. as already flushing 2023-04-15 02:53:52,493 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681527213260.c44270c3412c22e2c805f466aae6ff56. 2023-04-15 02:53:52,493 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-15 02:53:52,493 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-15 02:53:52,493 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:52,493 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-15 02:53:52,493 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34681,1681527209133' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-15 02:53:52,494 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:52,494 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:53:52,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:52,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:53:52,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:53:52,494 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:53:52,494 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-15 02:53:52,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:53:52,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:53:52,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:53:52,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:52,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:53:52,495 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34681,1681527209133' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-15 02:53:52,495 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-15 02:53:52,495 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@43b1de4c[Count = 0] remaining members to acquire global barrier 2023-04-15 02:53:52,495 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:53:52,496 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:53:52,496 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:53:52,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:53:52,496 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-15 02:53:52,496 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-15 02:53:52,496 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34681,1681527209133' in zk 2023-04-15 02:53:52,496 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:52,496 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-15 02:53:52,497 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:52,497 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-15 02:53:52,497 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-15 02:53:52,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:52,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:53:52,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:53:52,497 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-15 02:53:52,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:53:52,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:53:52,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:53:52,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:52,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:53:52,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:53:52,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:52,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34681,1681527209133': 2023-04-15 02:53:52,499 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34681,1681527209133' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-15 02:53:52,499 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-15 02:53:52,499 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-15 02:53:52,499 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-15 02:53:52,499 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-15 02:53:52,499 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-15 02:53:52,500 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:53:52,500 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:53:52,500 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:53:52,500 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-15 02:53:52,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:53:52,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:53:52,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:53:52,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:53:52,500 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:52,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:53:52,501 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-15 02:53:52,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-15 02:53:52,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:53:52,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:53:52,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:53:52,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:53:52,502 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:52,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:52,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:53:52,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:53:52,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:52,503 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-15 02:53:52,503 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-15 02:53:52,503 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:52,503 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-15 02:53:52,503 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-15 02:53:52,503 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:53:52,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-15 02:53:52,503 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:53:52,503 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:52,503 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:53:52,503 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:53:52,503 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-15 02:53:52,503 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-15 02:53:52,503 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-15 02:53:52,504 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-15 02:53:52,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-15 02:53:52,503 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:53:52,504 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-15 02:53:52,504 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-15 02:53:52,528 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=459 (bloomFilter=true), to=hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/c44270c3412c22e2c805f466aae6ff56/.tmp/C/0f371f625e8a4376b07e78dc48ad8518 2023-04-15 02:53:52,533 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/c44270c3412c22e2c805f466aae6ff56/.tmp/A/3e42386781194a9bab119952e28e6bba as hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/c44270c3412c22e2c805f466aae6ff56/A/3e42386781194a9bab119952e28e6bba 2023-04-15 02:53:52,537 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/c44270c3412c22e2c805f466aae6ff56/A/3e42386781194a9bab119952e28e6bba, entries=200, sequenceid=459, filesize=14.3 K 2023-04-15 02:53:52,538 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/c44270c3412c22e2c805f466aae6ff56/.tmp/B/0c1862f1233640a5810d6d90f85e4696 as hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/c44270c3412c22e2c805f466aae6ff56/B/0c1862f1233640a5810d6d90f85e4696 2023-04-15 02:53:52,543 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/c44270c3412c22e2c805f466aae6ff56/B/0c1862f1233640a5810d6d90f85e4696, entries=150, sequenceid=459, filesize=11.9 K 2023-04-15 02:53:52,544 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/c44270c3412c22e2c805f466aae6ff56/.tmp/C/0f371f625e8a4376b07e78dc48ad8518 as hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/c44270c3412c22e2c805f466aae6ff56/C/0f371f625e8a4376b07e78dc48ad8518 2023-04-15 02:53:52,550 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/c44270c3412c22e2c805f466aae6ff56/C/0f371f625e8a4376b07e78dc48ad8518, entries=150, sequenceid=459, filesize=11.9 K 2023-04-15 02:53:52,551 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=134.18 KB/137400 for c44270c3412c22e2c805f466aae6ff56 in 877ms, sequenceid=459, compaction requested=true 2023-04-15 02:53:52,551 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for c44270c3412c22e2c805f466aae6ff56: 2023-04-15 02:53:52,551 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-15 02:53:52,551 DEBUG [RS:0;jenkins-hbase20:34681-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-15 02:53:52,551 DEBUG [RS:0;jenkins-hbase20:34681-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-15 02:53:52,551 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-15 02:53:52,551 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-15 02:53:52,552 DEBUG [RS:0;jenkins-hbase20:34681-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37410 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-15 02:53:52,552 DEBUG [RS:0;jenkins-hbase20:34681-longCompactions-0] regionserver.HStore(1912): c44270c3412c22e2c805f466aae6ff56/B is initiating minor compaction (all files) 2023-04-15 02:53:52,552 INFO [RS:0;jenkins-hbase20:34681-longCompactions-0] regionserver.HRegion(2259): Starting compaction of c44270c3412c22e2c805f466aae6ff56/B in TestAcidGuarantees,,1681527213260.c44270c3412c22e2c805f466aae6ff56. 2023-04-15 02:53:52,553 INFO [RS:0;jenkins-hbase20:34681-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/c44270c3412c22e2c805f466aae6ff56/B/00a3a9b0abc448ae9c1f19b38c7583ff, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/c44270c3412c22e2c805f466aae6ff56/B/4eb9fadc993646b894e80997d6e745ca, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/c44270c3412c22e2c805f466aae6ff56/B/0c1862f1233640a5810d6d90f85e4696] into tmpdir=hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/c44270c3412c22e2c805f466aae6ff56/.tmp, totalSize=36.5 K 2023-04-15 02:53:52,557 DEBUG [RS:0;jenkins-hbase20:34681-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 39850 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-15 02:53:52,557 DEBUG [RS:0;jenkins-hbase20:34681-shortCompactions-0] regionserver.HStore(1912): c44270c3412c22e2c805f466aae6ff56/A is initiating minor compaction (all files) 2023-04-15 02:53:52,557 INFO [RS:0;jenkins-hbase20:34681-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of c44270c3412c22e2c805f466aae6ff56/A in TestAcidGuarantees,,1681527213260.c44270c3412c22e2c805f466aae6ff56. 2023-04-15 02:53:52,558 INFO [RS:0;jenkins-hbase20:34681-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/c44270c3412c22e2c805f466aae6ff56/A/58061441bf484cf5a57757d389a55706, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/c44270c3412c22e2c805f466aae6ff56/A/f5429821fb914e68ba7a97b8cf6ad6ad, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/c44270c3412c22e2c805f466aae6ff56/A/3e42386781194a9bab119952e28e6bba] into tmpdir=hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/c44270c3412c22e2c805f466aae6ff56/.tmp, totalSize=38.9 K 2023-04-15 02:53:52,558 DEBUG [RS:0;jenkins-hbase20:34681-longCompactions-0] compactions.Compactor(207): Compacting 00a3a9b0abc448ae9c1f19b38c7583ff, keycount=150, bloomtype=ROW, size=12.7 K, encoding=NONE, compression=NONE, seqNum=414, earliestPutTs=1681527229430 2023-04-15 02:53:52,558 DEBUG [RS:0;jenkins-hbase20:34681-shortCompactions-0] compactions.Compactor(207): Compacting 58061441bf484cf5a57757d389a55706, keycount=150, bloomtype=ROW, size=12.7 K, encoding=NONE, compression=NONE, seqNum=414, earliestPutTs=1681527229430 2023-04-15 02:53:52,558 DEBUG [RS:0;jenkins-hbase20:34681-longCompactions-0] compactions.Compactor(207): Compacting 4eb9fadc993646b894e80997d6e745ca, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=448, earliestPutTs=1681527229533 2023-04-15 02:53:52,558 DEBUG [RS:0;jenkins-hbase20:34681-shortCompactions-0] compactions.Compactor(207): Compacting f5429821fb914e68ba7a97b8cf6ad6ad, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=448, earliestPutTs=1681527229533 2023-04-15 02:53:52,558 DEBUG [RS:0;jenkins-hbase20:34681-longCompactions-0] compactions.Compactor(207): Compacting 0c1862f1233640a5810d6d90f85e4696, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=459, earliestPutTs=1681527231666 2023-04-15 02:53:52,559 DEBUG [RS:0;jenkins-hbase20:34681-shortCompactions-0] compactions.Compactor(207): Compacting 3e42386781194a9bab119952e28e6bba, keycount=200, bloomtype=ROW, size=14.3 K, encoding=NONE, compression=NONE, seqNum=459, earliestPutTs=1681527231666 2023-04-15 02:53:52,572 INFO [RS:0;jenkins-hbase20:34681-longCompactions-0] throttle.PressureAwareThroughputController(145): c44270c3412c22e2c805f466aae6ff56#B#compaction#84 average throughput is unlimited, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-15 02:53:52,574 INFO [RS:0;jenkins-hbase20:34681-shortCompactions-0] throttle.PressureAwareThroughputController(145): c44270c3412c22e2c805f466aae6ff56#A#compaction#85 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-15 02:53:52,591 DEBUG [RS:0;jenkins-hbase20:34681-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/c44270c3412c22e2c805f466aae6ff56/.tmp/A/7c3478a288494390bd0d3ef65f83a721 as hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/c44270c3412c22e2c805f466aae6ff56/A/7c3478a288494390bd0d3ef65f83a721 2023-04-15 02:53:52,601 INFO [RS:0;jenkins-hbase20:34681-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in c44270c3412c22e2c805f466aae6ff56/A of c44270c3412c22e2c805f466aae6ff56 into 7c3478a288494390bd0d3ef65f83a721(size=12.8 K), total size for store is 12.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-15 02:53:52,601 DEBUG [RS:0;jenkins-hbase20:34681-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for c44270c3412c22e2c805f466aae6ff56: 2023-04-15 02:53:52,602 INFO [RS:0;jenkins-hbase20:34681-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681527213260.c44270c3412c22e2c805f466aae6ff56., storeName=c44270c3412c22e2c805f466aae6ff56/A, priority=13, startTime=1681527232551; duration=0sec 2023-04-15 02:53:52,602 DEBUG [RS:0;jenkins-hbase20:34681-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-15 02:53:52,602 DEBUG [RS:0;jenkins-hbase20:34681-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-15 02:53:52,604 DEBUG [RS:0;jenkins-hbase20:34681-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37410 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-15 02:53:52,604 DEBUG [RS:0;jenkins-hbase20:34681-shortCompactions-0] regionserver.HStore(1912): c44270c3412c22e2c805f466aae6ff56/C is initiating minor compaction (all files) 2023-04-15 02:53:52,604 INFO [RS:0;jenkins-hbase20:34681-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of c44270c3412c22e2c805f466aae6ff56/C in TestAcidGuarantees,,1681527213260.c44270c3412c22e2c805f466aae6ff56. 2023-04-15 02:53:52,604 INFO [RS:0;jenkins-hbase20:34681-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/c44270c3412c22e2c805f466aae6ff56/C/4f1c63d6ddf542c6bd87f7fc14334ff0, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/c44270c3412c22e2c805f466aae6ff56/C/81b3e7d008db4b72a7cea39386d1e7a7, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/c44270c3412c22e2c805f466aae6ff56/C/0f371f625e8a4376b07e78dc48ad8518] into tmpdir=hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/c44270c3412c22e2c805f466aae6ff56/.tmp, totalSize=36.5 K 2023-04-15 02:53:52,604 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-15 02:53:52,605 DEBUG [RS:0;jenkins-hbase20:34681-shortCompactions-0] compactions.Compactor(207): Compacting 4f1c63d6ddf542c6bd87f7fc14334ff0, keycount=150, bloomtype=ROW, size=12.7 K, encoding=NONE, compression=NONE, seqNum=414, earliestPutTs=1681527229430 2023-04-15 02:53:52,605 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-15 02:53:52,605 DEBUG [RS:0;jenkins-hbase20:34681-shortCompactions-0] compactions.Compactor(207): Compacting 81b3e7d008db4b72a7cea39386d1e7a7, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=448, earliestPutTs=1681527229533 2023-04-15 02:53:52,605 DEBUG [RS:0;jenkins-hbase20:34681-shortCompactions-0] compactions.Compactor(207): Compacting 0f371f625e8a4376b07e78dc48ad8518, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=459, earliestPutTs=1681527231666 2023-04-15 02:53:52,606 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-15 02:53:52,607 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-15 02:53:52,607 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-15 02:53:52,607 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-15 02:53:52,607 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-15 02:53:52,607 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-15 02:53:52,607 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-15 02:53:52,608 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:53:52,608 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:53:52,608 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-15 02:53:52,608 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:52,608 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-15 02:53:52,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-15 02:53:52,609 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:52,609 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-15 02:53:52,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:53:52,609 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:53:52,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-15 02:53:52,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:53:52,609 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-15 02:53:52,610 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-15 02:53:52,610 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-15 02:53:52,610 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-15 02:53:52,610 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-15 02:53:52,610 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-15 02:53:52,610 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-15 02:53:52,610 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-15 02:53:52,610 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681527213260.c44270c3412c22e2c805f466aae6ff56. 2023-04-15 02:53:52,610 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681527213260.c44270c3412c22e2c805f466aae6ff56. started... 2023-04-15 02:53:52,610 INFO [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing c44270c3412c22e2c805f466aae6ff56 3/3 column families, dataSize=134.18 KB heapSize=375.61 KB 2023-04-15 02:53:52,610 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c44270c3412c22e2c805f466aae6ff56, store=A 2023-04-15 02:53:52,611 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-15 02:53:52,611 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c44270c3412c22e2c805f466aae6ff56, store=B 2023-04-15 02:53:52,611 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-15 02:53:52,611 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c44270c3412c22e2c805f466aae6ff56, store=C 2023-04-15 02:53:52,611 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-15 02:53:52,614 INFO [RS:0;jenkins-hbase20:34681-shortCompactions-0] throttle.PressureAwareThroughputController(145): c44270c3412c22e2c805f466aae6ff56#C#compaction#86 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-15 02:53:52,623 INFO [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=495 (bloomFilter=true), to=hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/c44270c3412c22e2c805f466aae6ff56/.tmp/A/4256873ddad14121a573f9103e7e2518 2023-04-15 02:53:52,633 INFO [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=495 (bloomFilter=true), to=hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/c44270c3412c22e2c805f466aae6ff56/.tmp/B/4a91b88ad17043e3abb471b5c4bd38f6 2023-04-15 02:53:52,649 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34681] regionserver.HRegion(9158): Flush requested on c44270c3412c22e2c805f466aae6ff56 2023-04-15 02:53:52,649 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681527213260.c44270c3412c22e2c805f466aae6ff56. as already flushing 2023-04-15 02:53:52,669 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c44270c3412c22e2c805f466aae6ff56, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:53:52,669 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34681] ipc.CallRunner(144): callId: 213 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:38530 deadline: 1681527292666, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c44270c3412c22e2c805f466aae6ff56, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:52,772 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c44270c3412c22e2c805f466aae6ff56, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:53:52,772 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34681] ipc.CallRunner(144): callId: 215 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:38530 deadline: 1681527292770, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c44270c3412c22e2c805f466aae6ff56, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:52,887 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c44270c3412c22e2c805f466aae6ff56, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:53:52,887 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34681] ipc.CallRunner(144): callId: 190 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:38504 deadline: 1681527292885, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c44270c3412c22e2c805f466aae6ff56, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:52,975 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c44270c3412c22e2c805f466aae6ff56, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:53:52,975 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34681] ipc.CallRunner(144): callId: 217 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:38530 deadline: 1681527292973, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c44270c3412c22e2c805f466aae6ff56, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:52,988 DEBUG [RS:0;jenkins-hbase20:34681-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/c44270c3412c22e2c805f466aae6ff56/.tmp/B/eb8fa00880054cf4b77f18772385523e as hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/c44270c3412c22e2c805f466aae6ff56/B/eb8fa00880054cf4b77f18772385523e 2023-04-15 02:53:52,994 INFO [RS:0;jenkins-hbase20:34681-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in c44270c3412c22e2c805f466aae6ff56/B of c44270c3412c22e2c805f466aae6ff56 into eb8fa00880054cf4b77f18772385523e(size=12.8 K), total size for store is 12.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-15 02:53:52,994 DEBUG [RS:0;jenkins-hbase20:34681-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for c44270c3412c22e2c805f466aae6ff56: 2023-04-15 02:53:52,994 INFO [RS:0;jenkins-hbase20:34681-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681527213260.c44270c3412c22e2c805f466aae6ff56., storeName=c44270c3412c22e2c805f466aae6ff56/B, priority=13, startTime=1681527232551; duration=0sec 2023-04-15 02:53:52,994 DEBUG [RS:0;jenkins-hbase20:34681-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-15 02:53:53,034 DEBUG [RS:0;jenkins-hbase20:34681-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/c44270c3412c22e2c805f466aae6ff56/.tmp/C/f41b5d2d34cc4d95ac9265fdc6c535f6 as hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/c44270c3412c22e2c805f466aae6ff56/C/f41b5d2d34cc4d95ac9265fdc6c535f6 2023-04-15 02:53:53,040 INFO [RS:0;jenkins-hbase20:34681-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in c44270c3412c22e2c805f466aae6ff56/C of c44270c3412c22e2c805f466aae6ff56 into f41b5d2d34cc4d95ac9265fdc6c535f6(size=12.8 K), total size for store is 12.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-15 02:53:53,040 DEBUG [RS:0;jenkins-hbase20:34681-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for c44270c3412c22e2c805f466aae6ff56: 2023-04-15 02:53:53,040 INFO [RS:0;jenkins-hbase20:34681-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681527213260.c44270c3412c22e2c805f466aae6ff56., storeName=c44270c3412c22e2c805f466aae6ff56/C, priority=13, startTime=1681527232551; duration=0sec 2023-04-15 02:53:53,040 DEBUG [RS:0;jenkins-hbase20:34681-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-15 02:53:53,043 INFO [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=495 (bloomFilter=true), to=hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/c44270c3412c22e2c805f466aae6ff56/.tmp/C/f9ce79cd57ec4075a6436a49d4b44314 2023-04-15 02:53:53,048 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/c44270c3412c22e2c805f466aae6ff56/.tmp/A/4256873ddad14121a573f9103e7e2518 as hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/c44270c3412c22e2c805f466aae6ff56/A/4256873ddad14121a573f9103e7e2518 2023-04-15 02:53:53,053 INFO [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/c44270c3412c22e2c805f466aae6ff56/A/4256873ddad14121a573f9103e7e2518, entries=150, sequenceid=495, filesize=11.9 K 2023-04-15 02:53:53,054 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/c44270c3412c22e2c805f466aae6ff56/.tmp/B/4a91b88ad17043e3abb471b5c4bd38f6 as hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/c44270c3412c22e2c805f466aae6ff56/B/4a91b88ad17043e3abb471b5c4bd38f6 2023-04-15 02:53:53,059 INFO [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/c44270c3412c22e2c805f466aae6ff56/B/4a91b88ad17043e3abb471b5c4bd38f6, entries=150, sequenceid=495, filesize=11.9 K 2023-04-15 02:53:53,060 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/c44270c3412c22e2c805f466aae6ff56/.tmp/C/f9ce79cd57ec4075a6436a49d4b44314 as hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/c44270c3412c22e2c805f466aae6ff56/C/f9ce79cd57ec4075a6436a49d4b44314 2023-04-15 02:53:53,065 INFO [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/c44270c3412c22e2c805f466aae6ff56/C/f9ce79cd57ec4075a6436a49d4b44314, entries=150, sequenceid=495, filesize=11.9 K 2023-04-15 02:53:53,066 INFO [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~134.18 KB/137400, heapSize ~375.56 KB/384576, currentSize=46.96 KB/48090 for c44270c3412c22e2c805f466aae6ff56 in 456ms, sequenceid=495, compaction requested=false 2023-04-15 02:53:53,066 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for c44270c3412c22e2c805f466aae6ff56: 2023-04-15 02:53:53,066 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681527213260.c44270c3412c22e2c805f466aae6ff56. 2023-04-15 02:53:53,066 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-15 02:53:53,066 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-15 02:53:53,066 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:53,069 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-15 02:53:53,069 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34681,1681527209133' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-15 02:53:53,070 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:53,070 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:53:53,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:53,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:53:53,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:53:53,071 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:53:53,071 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-15 02:53:53,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:53:53,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:53:53,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:53:53,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:53,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:53:53,072 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34681,1681527209133' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-15 02:53:53,072 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@59051c4f[Count = 0] remaining members to acquire global barrier 2023-04-15 02:53:53,072 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-15 02:53:53,072 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:53:53,073 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:53:53,073 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:53:53,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:53:53,073 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-15 02:53:53,073 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-15 02:53:53,073 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34681,1681527209133' in zk 2023-04-15 02:53:53,073 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:53,073 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-15 02:53:53,075 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:53,075 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-15 02:53:53,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:53,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:53:53,075 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-15 02:53:53,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:53:53,075 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-15 02:53:53,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:53:53,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:53:53,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:53:53,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:53,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:53:53,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:53:53,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:53,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34681,1681527209133': 2023-04-15 02:53:53,078 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34681,1681527209133' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-15 02:53:53,078 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-15 02:53:53,078 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-15 02:53:53,078 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-15 02:53:53,078 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-15 02:53:53,078 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-15 02:53:53,079 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:53:53,079 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:53:53,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:53:53,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:53:53,079 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:53:53,079 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-15 02:53:53,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:53:53,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:53:53,080 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-15 02:53:53,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:53:53,080 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:53,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-15 02:53:53,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:53:53,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:53:53,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:53:53,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:53:53,081 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:53,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:53,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:53:53,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:53:53,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:53,084 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:53,084 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:53:53,084 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-15 02:53:53,084 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-15 02:53:53,084 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-15 02:53:53,084 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-15 02:53:53,084 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-15 02:53:53,084 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-15 02:53:53,084 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-15 02:53:53,084 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:53:53,084 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:53,084 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:53:53,084 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:53:53,084 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-15 02:53:53,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-15 02:53:53,084 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-15 02:53:53,084 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:53:53,085 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-15 02:53:53,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-15 02:53:53,185 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-15 02:53:53,185 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-15 02:53:53,186 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-15 02:53:53,187 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-15 02:53:53,187 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-15 02:53:53,187 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-15 02:53:53,187 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-15 02:53:53,187 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-15 02:53:53,187 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-15 02:53:53,187 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:53:53,188 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:53:53,188 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-15 02:53:53,188 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:53,188 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-15 02:53:53,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-15 02:53:53,188 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:53,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:53:53,189 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-15 02:53:53,189 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:53:53,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-15 02:53:53,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:53:53,189 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-15 02:53:53,189 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-15 02:53:53,189 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-15 02:53:53,189 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-15 02:53:53,189 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-15 02:53:53,189 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-15 02:53:53,189 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-15 02:53:53,189 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-15 02:53:53,189 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681527213260.c44270c3412c22e2c805f466aae6ff56. 2023-04-15 02:53:53,189 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681527213260.c44270c3412c22e2c805f466aae6ff56. started... 2023-04-15 02:53:53,190 INFO [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing c44270c3412c22e2c805f466aae6ff56 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-15 02:53:53,190 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c44270c3412c22e2c805f466aae6ff56, store=A 2023-04-15 02:53:53,190 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-15 02:53:53,190 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c44270c3412c22e2c805f466aae6ff56, store=B 2023-04-15 02:53:53,190 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-15 02:53:53,190 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c44270c3412c22e2c805f466aae6ff56, store=C 2023-04-15 02:53:53,190 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-15 02:53:53,279 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34681] regionserver.HRegion(9158): Flush requested on c44270c3412c22e2c805f466aae6ff56 2023-04-15 02:53:53,279 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681527213260.c44270c3412c22e2c805f466aae6ff56. as already flushing 2023-04-15 02:53:53,321 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1681527233296, max timestamp=1681527233319Num uniques 859; 2023-04-15 02:53:53,321 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-3] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-04-15 02:53:53,322 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1681527233296, max timestamp=1681527233319Num uniques 859; 2023-04-15 02:53:53,322 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1681527233296, max timestamp=1681527233319Num uniques 859; 2023-04-15 02:53:53,322 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-6] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-04-15 02:53:53,322 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-04-15 02:53:53,356 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c44270c3412c22e2c805f466aae6ff56, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:53:53,356 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34681] ipc.CallRunner(144): callId: 251 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:38530 deadline: 1681527293354, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c44270c3412c22e2c805f466aae6ff56, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:53,458 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c44270c3412c22e2c805f466aae6ff56, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:53:53,458 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34681] ipc.CallRunner(144): callId: 253 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:38530 deadline: 1681527293457, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c44270c3412c22e2c805f466aae6ff56, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:53,601 INFO [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=507 (bloomFilter=true), to=hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/c44270c3412c22e2c805f466aae6ff56/.tmp/A/13ae9b1c4aac440fa3bf08f114450512 2023-04-15 02:53:53,659 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c44270c3412c22e2c805f466aae6ff56, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:53:53,660 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34681] ipc.CallRunner(144): callId: 255 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:38530 deadline: 1681527293659, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c44270c3412c22e2c805f466aae6ff56, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:53,890 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c44270c3412c22e2c805f466aae6ff56, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:53:53,891 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34681] ipc.CallRunner(144): callId: 192 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:38504 deadline: 1681527293890, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c44270c3412c22e2c805f466aae6ff56, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:53,964 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c44270c3412c22e2c805f466aae6ff56, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:53:53,964 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34681] ipc.CallRunner(144): callId: 257 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:38530 deadline: 1681527293963, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c44270c3412c22e2c805f466aae6ff56, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:54,012 INFO [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=507 (bloomFilter=true), to=hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/c44270c3412c22e2c805f466aae6ff56/.tmp/B/f27a2ce0602a4e448c49475f2b96bacd 2023-04-15 02:53:54,028 INFO [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=507 (bloomFilter=true), to=hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/c44270c3412c22e2c805f466aae6ff56/.tmp/C/fbbfb964ab7744078ba460854e9c539e 2023-04-15 02:53:54,033 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/c44270c3412c22e2c805f466aae6ff56/.tmp/A/13ae9b1c4aac440fa3bf08f114450512 as hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/c44270c3412c22e2c805f466aae6ff56/A/13ae9b1c4aac440fa3bf08f114450512 2023-04-15 02:53:54,037 INFO [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/c44270c3412c22e2c805f466aae6ff56/A/13ae9b1c4aac440fa3bf08f114450512, entries=150, sequenceid=507, filesize=11.9 K 2023-04-15 02:53:54,038 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/c44270c3412c22e2c805f466aae6ff56/.tmp/B/f27a2ce0602a4e448c49475f2b96bacd as hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/c44270c3412c22e2c805f466aae6ff56/B/f27a2ce0602a4e448c49475f2b96bacd 2023-04-15 02:53:54,043 INFO [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/c44270c3412c22e2c805f466aae6ff56/B/f27a2ce0602a4e448c49475f2b96bacd, entries=150, sequenceid=507, filesize=11.9 K 2023-04-15 02:53:54,044 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/c44270c3412c22e2c805f466aae6ff56/.tmp/C/fbbfb964ab7744078ba460854e9c539e as hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/c44270c3412c22e2c805f466aae6ff56/C/fbbfb964ab7744078ba460854e9c539e 2023-04-15 02:53:54,048 INFO [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/c44270c3412c22e2c805f466aae6ff56/C/fbbfb964ab7744078ba460854e9c539e, entries=150, sequenceid=507, filesize=11.9 K 2023-04-15 02:53:54,049 INFO [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=134.18 KB/137400 for c44270c3412c22e2c805f466aae6ff56 in 859ms, sequenceid=507, compaction requested=true 2023-04-15 02:53:54,049 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for c44270c3412c22e2c805f466aae6ff56: 2023-04-15 02:53:54,049 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681527213260.c44270c3412c22e2c805f466aae6ff56. 2023-04-15 02:53:54,049 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-15 02:53:54,049 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-15 02:53:54,049 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:54,050 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-15 02:53:54,050 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34681,1681527209133' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-15 02:53:54,051 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:53:54,051 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:54,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:54,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:53:54,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:53:54,052 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:53:54,052 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-15 02:53:54,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:53:54,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:53:54,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:53:54,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:54,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:53:54,053 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34681,1681527209133' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-15 02:53:54,053 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-15 02:53:54,053 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@169689ef[Count = 0] remaining members to acquire global barrier 2023-04-15 02:53:54,053 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:53:54,054 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:53:54,055 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:53:54,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:53:54,055 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-15 02:53:54,055 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-15 02:53:54,055 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34681,1681527209133' in zk 2023-04-15 02:53:54,055 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:54,055 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-15 02:53:54,055 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-15 02:53:54,056 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-15 02:53:54,056 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:54,056 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-15 02:53:54,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:54,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:53:54,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:53:54,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:53:54,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:53:54,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:53:54,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:54,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:53:54,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:53:54,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:54,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34681,1681527209133': 2023-04-15 02:53:54,058 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34681,1681527209133' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-15 02:53:54,058 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-15 02:53:54,058 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-15 02:53:54,058 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-15 02:53:54,058 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-15 02:53:54,058 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-15 02:53:54,059 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:53:54,059 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:53:54,059 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-15 02:53:54,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:53:54,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:53:54,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:53:54,059 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:53:54,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:53:54,059 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-15 02:53:54,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-15 02:53:54,060 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:54,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:53:54,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:53:54,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:53:54,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:53:54,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:53:54,061 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:54,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:54,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:53:54,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:53:54,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:54,063 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:54,063 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:53:54,063 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-15 02:53:54,063 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:53:54,063 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-15 02:53:54,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-15 02:53:54,063 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-15 02:53:54,063 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-15 02:53:54,063 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-15 02:53:54,063 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-15 02:53:54,063 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-15 02:53:54,063 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:54,063 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:53:54,063 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:53:54,063 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:53:54,064 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-15 02:53:54,064 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-15 02:53:54,064 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-15 02:53:54,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-15 02:53:54,164 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-15 02:53:54,164 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-15 02:53:54,165 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-15 02:53:54,166 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-15 02:53:54,166 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-15 02:53:54,166 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-15 02:53:54,166 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-15 02:53:54,166 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-15 02:53:54,166 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-15 02:53:54,167 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:53:54,167 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:53:54,167 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-15 02:53:54,167 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-15 02:53:54,167 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:54,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-15 02:53:54,168 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:54,168 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-15 02:53:54,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:53:54,168 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:53:54,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-15 02:53:54,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:53:54,168 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-15 02:53:54,168 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-15 02:53:54,168 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-15 02:53:54,168 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-15 02:53:54,169 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-15 02:53:54,169 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-15 02:53:54,169 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-15 02:53:54,169 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-15 02:53:54,169 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681527213260.c44270c3412c22e2c805f466aae6ff56. 2023-04-15 02:53:54,169 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681527213260.c44270c3412c22e2c805f466aae6ff56. started... 2023-04-15 02:53:54,169 INFO [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing c44270c3412c22e2c805f466aae6ff56 3/3 column families, dataSize=134.18 KB heapSize=375.61 KB 2023-04-15 02:53:54,169 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c44270c3412c22e2c805f466aae6ff56, store=A 2023-04-15 02:53:54,169 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-15 02:53:54,169 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c44270c3412c22e2c805f466aae6ff56, store=B 2023-04-15 02:53:54,169 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-15 02:53:54,169 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c44270c3412c22e2c805f466aae6ff56, store=C 2023-04-15 02:53:54,169 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-15 02:53:54,468 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34681] regionserver.HRegion(9158): Flush requested on c44270c3412c22e2c805f466aae6ff56 2023-04-15 02:53:54,468 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681527213260.c44270c3412c22e2c805f466aae6ff56. as already flushing 2023-04-15 02:53:54,489 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c44270c3412c22e2c805f466aae6ff56, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:53:54,489 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34681] ipc.CallRunner(144): callId: 266 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:38530 deadline: 1681527294485, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c44270c3412c22e2c805f466aae6ff56, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:54,582 INFO [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=542 (bloomFilter=true), to=hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/c44270c3412c22e2c805f466aae6ff56/.tmp/A/47d9637ccbe249a8a1c045372007b248 2023-04-15 02:53:54,590 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c44270c3412c22e2c805f466aae6ff56, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:53:54,590 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34681] ipc.CallRunner(144): callId: 268 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:38530 deadline: 1681527294590, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c44270c3412c22e2c805f466aae6ff56, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:54,793 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c44270c3412c22e2c805f466aae6ff56, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:53:54,793 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34681] ipc.CallRunner(144): callId: 270 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:38530 deadline: 1681527294792, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c44270c3412c22e2c805f466aae6ff56, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:54,994 INFO [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=542 (bloomFilter=true), to=hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/c44270c3412c22e2c805f466aae6ff56/.tmp/B/ed854dd66ac04f8fbefcf4e10188adb1 2023-04-15 02:53:55,099 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c44270c3412c22e2c805f466aae6ff56, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:53:55,099 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34681] ipc.CallRunner(144): callId: 272 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:38530 deadline: 1681527295096, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c44270c3412c22e2c805f466aae6ff56, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:55,404 INFO [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=542 (bloomFilter=true), to=hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/c44270c3412c22e2c805f466aae6ff56/.tmp/C/b4c740841db14dbe87104caa2ce01425 2023-04-15 02:53:55,411 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/c44270c3412c22e2c805f466aae6ff56/.tmp/A/47d9637ccbe249a8a1c045372007b248 as hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/c44270c3412c22e2c805f466aae6ff56/A/47d9637ccbe249a8a1c045372007b248 2023-04-15 02:53:55,415 INFO [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/c44270c3412c22e2c805f466aae6ff56/A/47d9637ccbe249a8a1c045372007b248, entries=150, sequenceid=542, filesize=11.9 K 2023-04-15 02:53:55,416 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/c44270c3412c22e2c805f466aae6ff56/.tmp/B/ed854dd66ac04f8fbefcf4e10188adb1 as hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/c44270c3412c22e2c805f466aae6ff56/B/ed854dd66ac04f8fbefcf4e10188adb1 2023-04-15 02:53:55,420 INFO [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/c44270c3412c22e2c805f466aae6ff56/B/ed854dd66ac04f8fbefcf4e10188adb1, entries=150, sequenceid=542, filesize=11.9 K 2023-04-15 02:53:55,422 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/c44270c3412c22e2c805f466aae6ff56/.tmp/C/b4c740841db14dbe87104caa2ce01425 as hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/c44270c3412c22e2c805f466aae6ff56/C/b4c740841db14dbe87104caa2ce01425 2023-04-15 02:53:55,431 INFO [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/c44270c3412c22e2c805f466aae6ff56/C/b4c740841db14dbe87104caa2ce01425, entries=150, sequenceid=542, filesize=11.9 K 2023-04-15 02:53:55,431 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.StoreScanner(961): StoreScanner already closing. There is no need to updateReaders 2023-04-15 02:53:55,432 INFO [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~134.18 KB/137400, heapSize ~375.56 KB/384576, currentSize=46.96 KB/48090 for c44270c3412c22e2c805f466aae6ff56 in 1263ms, sequenceid=542, compaction requested=true 2023-04-15 02:53:55,432 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for c44270c3412c22e2c805f466aae6ff56: 2023-04-15 02:53:55,432 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681527213260.c44270c3412c22e2c805f466aae6ff56. 2023-04-15 02:53:55,432 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-15 02:53:55,432 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-15 02:53:55,432 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:55,435 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-15 02:53:55,435 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34681,1681527209133' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-15 02:53:55,436 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:53:55,436 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:55,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:55,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:53:55,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:53:55,436 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:53:55,436 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-15 02:53:55,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:53:55,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:53:55,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:53:55,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:55,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:53:55,437 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34681,1681527209133' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-15 02:53:55,438 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3f85f26c[Count = 0] remaining members to acquire global barrier 2023-04-15 02:53:55,438 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-15 02:53:55,438 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:53:55,438 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:53:55,439 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:53:55,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:53:55,439 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-15 02:53:55,439 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:55,439 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-15 02:53:55,439 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-15 02:53:55,439 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34681,1681527209133' in zk 2023-04-15 02:53:55,439 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:55,439 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-15 02:53:55,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:55,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:53:55,440 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-15 02:53:55,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:53:55,440 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-15 02:53:55,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:53:55,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:53:55,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:53:55,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:55,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:53:55,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:53:55,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:55,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34681,1681527209133': 2023-04-15 02:53:55,442 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34681,1681527209133' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-15 02:53:55,442 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-15 02:53:55,442 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-15 02:53:55,442 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-15 02:53:55,442 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-15 02:53:55,442 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-15 02:53:55,443 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:53:55,443 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:53:55,443 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:53:55,443 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-15 02:53:55,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:53:55,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:53:55,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:53:55,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:53:55,443 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-15 02:53:55,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-15 02:53:55,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:53:55,444 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:55,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:53:55,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:53:55,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:53:55,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:53:55,445 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:55,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:55,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:53:55,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:53:55,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:55,446 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:55,446 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:53:55,446 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-15 02:53:55,446 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:53:55,447 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-15 02:53:55,447 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-15 02:53:55,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-15 02:53:55,447 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:55,447 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:53:55,447 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-15 02:53:55,447 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:53:55,447 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:53:55,447 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-15 02:53:55,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-15 02:53:55,450 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-15 02:53:55,450 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-15 02:53:55,450 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-15 02:53:55,450 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-15 02:53:55,450 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-15 02:53:55,535 DEBUG [Thread-152] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x046b5e40 to 127.0.0.1:51762 2023-04-15 02:53:55,535 DEBUG [Thread-152] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-15 02:53:55,537 DEBUG [Thread-154] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x127a87ef to 127.0.0.1:51762 2023-04-15 02:53:55,537 DEBUG [Thread-154] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-15 02:53:55,542 DEBUG [Thread-156] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x19e3f14f to 127.0.0.1:51762 2023-04-15 02:53:55,542 DEBUG [Thread-156] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-15 02:53:55,543 DEBUG [Thread-158] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x38ebb9f9 to 127.0.0.1:51762 2023-04-15 02:53:55,543 DEBUG [Thread-158] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-15 02:53:55,550 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-15 02:53:55,551 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-15 02:53:55,605 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34681] regionserver.HRegion(9158): Flush requested on c44270c3412c22e2c805f466aae6ff56 2023-04-15 02:53:55,605 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing c44270c3412c22e2c805f466aae6ff56 3/3 column families, dataSize=53.67 KB heapSize=160.13 KB 2023-04-15 02:53:55,606 DEBUG [Thread-147] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x13e837e0 to 127.0.0.1:51762 2023-04-15 02:53:55,606 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c44270c3412c22e2c805f466aae6ff56, store=A 2023-04-15 02:53:55,606 DEBUG [Thread-147] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-15 02:53:55,606 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-15 02:53:55,606 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c44270c3412c22e2c805f466aae6ff56, store=B 2023-04-15 02:53:55,607 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-15 02:53:55,607 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c44270c3412c22e2c805f466aae6ff56, store=C 2023-04-15 02:53:55,607 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-15 02:53:55,634 DEBUG [Thread-143] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x05be6782 to 127.0.0.1:51762 2023-04-15 02:53:55,634 DEBUG [Thread-143] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-15 02:53:55,875 DEBUG [Thread-149] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x3167991b to 127.0.0.1:51762 2023-04-15 02:53:55,875 DEBUG [Thread-149] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-15 02:53:55,889 DEBUG [Thread-145] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x5610151a to 127.0.0.1:51762 2023-04-15 02:53:55,889 DEBUG [Thread-145] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-15 02:53:55,899 DEBUG [Thread-141] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x346ea1a1 to 127.0.0.1:51762 2023-04-15 02:53:55,899 DEBUG [Thread-141] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-15 02:53:55,900 INFO [Listener at localhost.localdomain/33693] hbase.AcidGuaranteesTestTool(392): Finished test. Writers: 2023-04-15 02:53:55,900 INFO [Listener at localhost.localdomain/33693] hbase.AcidGuaranteesTestTool(394): wrote 120 2023-04-15 02:53:55,900 INFO [Listener at localhost.localdomain/33693] hbase.AcidGuaranteesTestTool(394): wrote 23 2023-04-15 02:53:55,900 INFO [Listener at localhost.localdomain/33693] hbase.AcidGuaranteesTestTool(394): wrote 71 2023-04-15 02:53:55,901 INFO [Listener at localhost.localdomain/33693] hbase.AcidGuaranteesTestTool(394): wrote 174 2023-04-15 02:53:55,901 INFO [Listener at localhost.localdomain/33693] hbase.AcidGuaranteesTestTool(394): wrote 66 2023-04-15 02:53:55,901 INFO [Listener at localhost.localdomain/33693] hbase.AcidGuaranteesTestTool(396): Readers: 2023-04-15 02:53:55,901 INFO [Listener at localhost.localdomain/33693] hbase.AcidGuaranteesTestTool(398): read 5692 2023-04-15 02:53:55,901 INFO [Listener at localhost.localdomain/33693] hbase.AcidGuaranteesTestTool(398): read 5419 2023-04-15 02:53:55,901 INFO [Listener at localhost.localdomain/33693] hbase.AcidGuaranteesTestTool(400): Scanners: 2023-04-15 02:53:55,901 INFO [Listener at localhost.localdomain/33693] hbase.AcidGuaranteesTestTool(402): scanned 2384 2023-04-15 02:53:55,901 INFO [Listener at localhost.localdomain/33693] hbase.AcidGuaranteesTestTool(403): verified 7149 rows 2023-04-15 02:53:55,902 INFO [Listener at localhost.localdomain/33693] hbase.AcidGuaranteesTestTool(402): scanned 2393 2023-04-15 02:53:55,902 INFO [Listener at localhost.localdomain/33693] hbase.AcidGuaranteesTestTool(403): verified 7176 rows 2023-04-15 02:53:55,902 INFO [Listener at localhost.localdomain/33693] client.ConnectionImplementation(1974): Closing master protocol: MasterService 2023-04-15 02:53:55,902 DEBUG [Listener at localhost.localdomain/33693] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x7f4385a1 to 127.0.0.1:51762 2023-04-15 02:53:55,902 DEBUG [Listener at localhost.localdomain/33693] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-15 02:53:55,907 INFO [Listener at localhost.localdomain/33693] client.HBaseAdmin$15(890): Started disable of TestAcidGuarantees 2023-04-15 02:53:55,912 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.HMaster$11(2418): Client=jenkins//148.251.75.209 disable TestAcidGuarantees 2023-04-15 02:53:55,918 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] procedure2.ProcedureExecutor(1029): Stored pid=12, state=RUNNABLE:DISABLE_TABLE_PREPARE; DisableTableProcedure table=TestAcidGuarantees 2023-04-15 02:53:55,921 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1681527235921"}]},"ts":"1681527235921"} 2023-04-15 02:53:55,922 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(1227): Checking to see if procedure is done pid=12 2023-04-15 02:53:55,922 INFO [PEWorker-3] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=DISABLING in hbase:meta 2023-04-15 02:53:55,923 INFO [PEWorker-3] procedure.DisableTableProcedure(293): Set TestAcidGuarantees to state=DISABLING 2023-04-15 02:53:55,925 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=13, ppid=12, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=c44270c3412c22e2c805f466aae6ff56, UNASSIGN}] 2023-04-15 02:53:55,925 INFO [PEWorker-3] procedure.MasterProcedureScheduler(727): Took xlock for pid=13, ppid=12, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=c44270c3412c22e2c805f466aae6ff56, UNASSIGN 2023-04-15 02:53:55,926 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=13 updating hbase:meta row=c44270c3412c22e2c805f466aae6ff56, regionState=CLOSING, regionLocation=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:53:55,926 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1681527213260.c44270c3412c22e2c805f466aae6ff56.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1681527235926"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1681527235926"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1681527235926"}]},"ts":"1681527235926"} 2023-04-15 02:53:55,928 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=14, ppid=13, state=RUNNABLE; CloseRegionProcedure c44270c3412c22e2c805f466aae6ff56, server=jenkins-hbase20.apache.org,34681,1681527209133}] 2023-04-15 02:53:56,019 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=553 (bloomFilter=true), to=hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/c44270c3412c22e2c805f466aae6ff56/.tmp/A/2aaabf9165124956aa0d996a968ab13c 2023-04-15 02:53:56,023 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(1227): Checking to see if procedure is done pid=12 2023-04-15 02:53:56,089 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] handler.UnassignRegionHandler(109): Close c44270c3412c22e2c805f466aae6ff56 2023-04-15 02:53:56,090 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1604): Closing c44270c3412c22e2c805f466aae6ff56, disabling compactions & flushes 2023-04-15 02:53:56,090 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1858): waiting for 0 compactions & cache flush to complete for region TestAcidGuarantees,,1681527213260.c44270c3412c22e2c805f466aae6ff56. 2023-04-15 02:53:56,225 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(1227): Checking to see if procedure is done pid=12 2023-04-15 02:53:56,435 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=553 (bloomFilter=true), to=hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/c44270c3412c22e2c805f466aae6ff56/.tmp/B/9969523cf245497292e9adb1cd9fb873 2023-04-15 02:53:56,527 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(1227): Checking to see if procedure is done pid=12 2023-04-15 02:53:56,854 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=553 (bloomFilter=true), to=hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/c44270c3412c22e2c805f466aae6ff56/.tmp/C/b605227d5ffe4ef8b2da2e8de095324e 2023-04-15 02:53:56,867 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/c44270c3412c22e2c805f466aae6ff56/.tmp/A/2aaabf9165124956aa0d996a968ab13c as hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/c44270c3412c22e2c805f466aae6ff56/A/2aaabf9165124956aa0d996a968ab13c 2023-04-15 02:53:56,872 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/c44270c3412c22e2c805f466aae6ff56/A/2aaabf9165124956aa0d996a968ab13c, entries=150, sequenceid=553, filesize=11.9 K 2023-04-15 02:53:56,873 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/c44270c3412c22e2c805f466aae6ff56/.tmp/B/9969523cf245497292e9adb1cd9fb873 as hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/c44270c3412c22e2c805f466aae6ff56/B/9969523cf245497292e9adb1cd9fb873 2023-04-15 02:53:56,877 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/c44270c3412c22e2c805f466aae6ff56/B/9969523cf245497292e9adb1cd9fb873, entries=150, sequenceid=553, filesize=11.9 K 2023-04-15 02:53:56,877 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/c44270c3412c22e2c805f466aae6ff56/.tmp/C/b605227d5ffe4ef8b2da2e8de095324e as hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/c44270c3412c22e2c805f466aae6ff56/C/b605227d5ffe4ef8b2da2e8de095324e 2023-04-15 02:53:56,881 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/c44270c3412c22e2c805f466aae6ff56/C/b605227d5ffe4ef8b2da2e8de095324e, entries=150, sequenceid=553, filesize=11.9 K 2023-04-15 02:53:56,882 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=26.84 KB/27480 for c44270c3412c22e2c805f466aae6ff56 in 1277ms, sequenceid=553, compaction requested=true 2023-04-15 02:53:56,882 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for c44270c3412c22e2c805f466aae6ff56: 2023-04-15 02:53:56,882 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1681527213260.c44270c3412c22e2c805f466aae6ff56. 2023-04-15 02:53:56,882 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1681527213260.c44270c3412c22e2c805f466aae6ff56. 2023-04-15 02:53:56,882 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-15 02:53:56,882 DEBUG [RS:0;jenkins-hbase20:34681-shortCompactions-0] regionserver.CompactSplit(415): Not compacting TestAcidGuarantees,,1681527213260.c44270c3412c22e2c805f466aae6ff56. because compaction request was cancelled 2023-04-15 02:53:56,883 DEBUG [RS:0;jenkins-hbase20:34681-longCompactions-0] regionserver.CompactSplit(415): Not compacting TestAcidGuarantees,,1681527213260.c44270c3412c22e2c805f466aae6ff56. because compaction request was cancelled 2023-04-15 02:53:56,882 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1681527213260.c44270c3412c22e2c805f466aae6ff56. after waiting 0 ms 2023-04-15 02:53:56,882 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-15 02:53:56,883 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-15 02:53:56,883 DEBUG [RS:0;jenkins-hbase20:34681-shortCompactions-0] regionserver.CompactSplit(415): Not compacting TestAcidGuarantees,,1681527213260.c44270c3412c22e2c805f466aae6ff56. because compaction request was cancelled 2023-04-15 02:53:56,883 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1681527213260.c44270c3412c22e2c805f466aae6ff56. 2023-04-15 02:53:56,883 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(2745): Flushing c44270c3412c22e2c805f466aae6ff56 3/3 column families, dataSize=26.84 KB heapSize=80.44 KB 2023-04-15 02:53:56,883 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c44270c3412c22e2c805f466aae6ff56, store=A 2023-04-15 02:53:56,883 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-15 02:53:56,883 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c44270c3412c22e2c805f466aae6ff56, store=B 2023-04-15 02:53:56,883 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-15 02:53:56,883 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c44270c3412c22e2c805f466aae6ff56, store=C 2023-04-15 02:53:56,883 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-15 02:53:57,029 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(1227): Checking to see if procedure is done pid=12 2023-04-15 02:53:57,291 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=560 (bloomFilter=true), to=hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/c44270c3412c22e2c805f466aae6ff56/.tmp/A/67683853907140e0b4f8bebc8daa4dba 2023-04-15 02:53:57,711 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=560 (bloomFilter=true), to=hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/c44270c3412c22e2c805f466aae6ff56/.tmp/B/d1d472710d0c479aaa793ef456adc49f 2023-04-15 02:53:58,030 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(1227): Checking to see if procedure is done pid=12 2023-04-15 02:53:58,133 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=560 (bloomFilter=true), to=hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/c44270c3412c22e2c805f466aae6ff56/.tmp/C/7b936adb3bbb40e4ad1e9ee9845861ae 2023-04-15 02:53:58,144 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/c44270c3412c22e2c805f466aae6ff56/.tmp/A/67683853907140e0b4f8bebc8daa4dba as hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/c44270c3412c22e2c805f466aae6ff56/A/67683853907140e0b4f8bebc8daa4dba 2023-04-15 02:53:58,149 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/c44270c3412c22e2c805f466aae6ff56/A/67683853907140e0b4f8bebc8daa4dba, entries=50, sequenceid=560, filesize=7.1 K 2023-04-15 02:53:58,150 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/c44270c3412c22e2c805f466aae6ff56/.tmp/B/d1d472710d0c479aaa793ef456adc49f as hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/c44270c3412c22e2c805f466aae6ff56/B/d1d472710d0c479aaa793ef456adc49f 2023-04-15 02:53:58,156 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/c44270c3412c22e2c805f466aae6ff56/B/d1d472710d0c479aaa793ef456adc49f, entries=50, sequenceid=560, filesize=7.1 K 2023-04-15 02:53:58,156 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/c44270c3412c22e2c805f466aae6ff56/.tmp/C/7b936adb3bbb40e4ad1e9ee9845861ae as hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/c44270c3412c22e2c805f466aae6ff56/C/7b936adb3bbb40e4ad1e9ee9845861ae 2023-04-15 02:53:58,162 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/c44270c3412c22e2c805f466aae6ff56/C/7b936adb3bbb40e4ad1e9ee9845861ae, entries=50, sequenceid=560, filesize=7.1 K 2023-04-15 02:53:58,163 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(2948): Finished flush of dataSize ~26.84 KB/27480, heapSize ~80.39 KB/82320, currentSize=0 B/0 for c44270c3412c22e2c805f466aae6ff56 in 1280ms, sequenceid=560, compaction requested=true 2023-04-15 02:53:58,180 DEBUG [StoreCloser-TestAcidGuarantees,,1681527213260.c44270c3412c22e2c805f466aae6ff56.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/c44270c3412c22e2c805f466aae6ff56/A/acacce0fff2a4181b0d8c913048e49e0, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/c44270c3412c22e2c805f466aae6ff56/A/6dd7dd3e6325495f90a18096bd9a44a8, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/c44270c3412c22e2c805f466aae6ff56/A/4dee2a8a76104fdc88dd783f0a5f1aa1, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/c44270c3412c22e2c805f466aae6ff56/A/3bbb13c5de744a0fa342f72bc2c1221f, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/c44270c3412c22e2c805f466aae6ff56/A/a7228edc07f64d3d8590a75e2dfac85a, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/c44270c3412c22e2c805f466aae6ff56/A/c6471c87fd14400580d23d2501310834, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/c44270c3412c22e2c805f466aae6ff56/A/1f795105d9aa47a4b295f5e05d7a5e03, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/c44270c3412c22e2c805f466aae6ff56/A/94a6ea33c5964300bba31cc9d9175e53, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/c44270c3412c22e2c805f466aae6ff56/A/cc98a063c4a940448766c211027bf809, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/c44270c3412c22e2c805f466aae6ff56/A/2f990a41b18641609c1dda35b612f63d, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/c44270c3412c22e2c805f466aae6ff56/A/a4e8adecaa744cf8bdc2810ae3aac070, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/c44270c3412c22e2c805f466aae6ff56/A/8f7b03202b8040ad85cab6a1a5522b63, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/c44270c3412c22e2c805f466aae6ff56/A/738be694a3d34996a54af27f4b8b76f0, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/c44270c3412c22e2c805f466aae6ff56/A/3fecd097e4dc418788d8e03e486d995f, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/c44270c3412c22e2c805f466aae6ff56/A/90066fdb8faf41cd90972274f2085d6c, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/c44270c3412c22e2c805f466aae6ff56/A/ae40346c39e94bcdb9eb81fbc0a744f2, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/c44270c3412c22e2c805f466aae6ff56/A/c0f7d1d451b94af2b158f512a2c089ed, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/c44270c3412c22e2c805f466aae6ff56/A/9b3a32d29fc74086b895f2308fd5a598, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/c44270c3412c22e2c805f466aae6ff56/A/6f6754556026426b882980c6173c34d5, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/c44270c3412c22e2c805f466aae6ff56/A/107ad8c17894439e9194631ffc9b3c4d, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/c44270c3412c22e2c805f466aae6ff56/A/ff96f2210e8741199ea6a6a64e0596ad, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/c44270c3412c22e2c805f466aae6ff56/A/c14d9273a0a44d31ac175cdf798bf17b, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/c44270c3412c22e2c805f466aae6ff56/A/2573076332df46de97607613d26daddd, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/c44270c3412c22e2c805f466aae6ff56/A/3a38a999ff8441b7b08fcc09067aca06, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/c44270c3412c22e2c805f466aae6ff56/A/58061441bf484cf5a57757d389a55706, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/c44270c3412c22e2c805f466aae6ff56/A/9adad4a9679d4598acc5a03552f1531e, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/c44270c3412c22e2c805f466aae6ff56/A/f5429821fb914e68ba7a97b8cf6ad6ad, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/c44270c3412c22e2c805f466aae6ff56/A/3e42386781194a9bab119952e28e6bba] to archive 2023-04-15 02:53:58,181 DEBUG [StoreCloser-TestAcidGuarantees,,1681527213260.c44270c3412c22e2c805f466aae6ff56.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-15 02:53:58,186 DEBUG [StoreCloser-TestAcidGuarantees,,1681527213260.c44270c3412c22e2c805f466aae6ff56.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/c44270c3412c22e2c805f466aae6ff56/A/acacce0fff2a4181b0d8c913048e49e0 to hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/archive/data/default/TestAcidGuarantees/c44270c3412c22e2c805f466aae6ff56/A/acacce0fff2a4181b0d8c913048e49e0 2023-04-15 02:53:58,187 DEBUG [StoreCloser-TestAcidGuarantees,,1681527213260.c44270c3412c22e2c805f466aae6ff56.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/c44270c3412c22e2c805f466aae6ff56/A/6dd7dd3e6325495f90a18096bd9a44a8 to hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/archive/data/default/TestAcidGuarantees/c44270c3412c22e2c805f466aae6ff56/A/6dd7dd3e6325495f90a18096bd9a44a8 2023-04-15 02:53:58,188 DEBUG [StoreCloser-TestAcidGuarantees,,1681527213260.c44270c3412c22e2c805f466aae6ff56.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/c44270c3412c22e2c805f466aae6ff56/A/4dee2a8a76104fdc88dd783f0a5f1aa1 to hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/archive/data/default/TestAcidGuarantees/c44270c3412c22e2c805f466aae6ff56/A/4dee2a8a76104fdc88dd783f0a5f1aa1 2023-04-15 02:53:58,189 DEBUG [StoreCloser-TestAcidGuarantees,,1681527213260.c44270c3412c22e2c805f466aae6ff56.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/c44270c3412c22e2c805f466aae6ff56/A/3bbb13c5de744a0fa342f72bc2c1221f to hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/archive/data/default/TestAcidGuarantees/c44270c3412c22e2c805f466aae6ff56/A/3bbb13c5de744a0fa342f72bc2c1221f 2023-04-15 02:53:58,190 DEBUG [StoreCloser-TestAcidGuarantees,,1681527213260.c44270c3412c22e2c805f466aae6ff56.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/c44270c3412c22e2c805f466aae6ff56/A/a7228edc07f64d3d8590a75e2dfac85a to hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/archive/data/default/TestAcidGuarantees/c44270c3412c22e2c805f466aae6ff56/A/a7228edc07f64d3d8590a75e2dfac85a 2023-04-15 02:53:58,191 DEBUG [StoreCloser-TestAcidGuarantees,,1681527213260.c44270c3412c22e2c805f466aae6ff56.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/c44270c3412c22e2c805f466aae6ff56/A/c6471c87fd14400580d23d2501310834 to hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/archive/data/default/TestAcidGuarantees/c44270c3412c22e2c805f466aae6ff56/A/c6471c87fd14400580d23d2501310834 2023-04-15 02:53:58,192 DEBUG [StoreCloser-TestAcidGuarantees,,1681527213260.c44270c3412c22e2c805f466aae6ff56.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/c44270c3412c22e2c805f466aae6ff56/A/1f795105d9aa47a4b295f5e05d7a5e03 to hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/archive/data/default/TestAcidGuarantees/c44270c3412c22e2c805f466aae6ff56/A/1f795105d9aa47a4b295f5e05d7a5e03 2023-04-15 02:53:58,193 DEBUG [StoreCloser-TestAcidGuarantees,,1681527213260.c44270c3412c22e2c805f466aae6ff56.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/c44270c3412c22e2c805f466aae6ff56/A/94a6ea33c5964300bba31cc9d9175e53 to hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/archive/data/default/TestAcidGuarantees/c44270c3412c22e2c805f466aae6ff56/A/94a6ea33c5964300bba31cc9d9175e53 2023-04-15 02:53:58,194 DEBUG [StoreCloser-TestAcidGuarantees,,1681527213260.c44270c3412c22e2c805f466aae6ff56.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/c44270c3412c22e2c805f466aae6ff56/A/cc98a063c4a940448766c211027bf809 to hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/archive/data/default/TestAcidGuarantees/c44270c3412c22e2c805f466aae6ff56/A/cc98a063c4a940448766c211027bf809 2023-04-15 02:53:58,195 DEBUG [StoreCloser-TestAcidGuarantees,,1681527213260.c44270c3412c22e2c805f466aae6ff56.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/c44270c3412c22e2c805f466aae6ff56/A/2f990a41b18641609c1dda35b612f63d to hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/archive/data/default/TestAcidGuarantees/c44270c3412c22e2c805f466aae6ff56/A/2f990a41b18641609c1dda35b612f63d 2023-04-15 02:53:58,196 DEBUG [StoreCloser-TestAcidGuarantees,,1681527213260.c44270c3412c22e2c805f466aae6ff56.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/c44270c3412c22e2c805f466aae6ff56/A/a4e8adecaa744cf8bdc2810ae3aac070 to hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/archive/data/default/TestAcidGuarantees/c44270c3412c22e2c805f466aae6ff56/A/a4e8adecaa744cf8bdc2810ae3aac070 2023-04-15 02:53:58,197 DEBUG [StoreCloser-TestAcidGuarantees,,1681527213260.c44270c3412c22e2c805f466aae6ff56.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/c44270c3412c22e2c805f466aae6ff56/A/8f7b03202b8040ad85cab6a1a5522b63 to hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/archive/data/default/TestAcidGuarantees/c44270c3412c22e2c805f466aae6ff56/A/8f7b03202b8040ad85cab6a1a5522b63 2023-04-15 02:53:58,198 DEBUG [StoreCloser-TestAcidGuarantees,,1681527213260.c44270c3412c22e2c805f466aae6ff56.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/c44270c3412c22e2c805f466aae6ff56/A/738be694a3d34996a54af27f4b8b76f0 to hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/archive/data/default/TestAcidGuarantees/c44270c3412c22e2c805f466aae6ff56/A/738be694a3d34996a54af27f4b8b76f0 2023-04-15 02:53:58,200 DEBUG [StoreCloser-TestAcidGuarantees,,1681527213260.c44270c3412c22e2c805f466aae6ff56.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/c44270c3412c22e2c805f466aae6ff56/A/3fecd097e4dc418788d8e03e486d995f to hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/archive/data/default/TestAcidGuarantees/c44270c3412c22e2c805f466aae6ff56/A/3fecd097e4dc418788d8e03e486d995f 2023-04-15 02:53:58,201 DEBUG [StoreCloser-TestAcidGuarantees,,1681527213260.c44270c3412c22e2c805f466aae6ff56.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/c44270c3412c22e2c805f466aae6ff56/A/90066fdb8faf41cd90972274f2085d6c to hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/archive/data/default/TestAcidGuarantees/c44270c3412c22e2c805f466aae6ff56/A/90066fdb8faf41cd90972274f2085d6c 2023-04-15 02:53:58,203 DEBUG [StoreCloser-TestAcidGuarantees,,1681527213260.c44270c3412c22e2c805f466aae6ff56.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/c44270c3412c22e2c805f466aae6ff56/A/ae40346c39e94bcdb9eb81fbc0a744f2 to hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/archive/data/default/TestAcidGuarantees/c44270c3412c22e2c805f466aae6ff56/A/ae40346c39e94bcdb9eb81fbc0a744f2 2023-04-15 02:53:58,205 DEBUG [StoreCloser-TestAcidGuarantees,,1681527213260.c44270c3412c22e2c805f466aae6ff56.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/c44270c3412c22e2c805f466aae6ff56/A/c0f7d1d451b94af2b158f512a2c089ed to hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/archive/data/default/TestAcidGuarantees/c44270c3412c22e2c805f466aae6ff56/A/c0f7d1d451b94af2b158f512a2c089ed 2023-04-15 02:53:58,206 DEBUG [StoreCloser-TestAcidGuarantees,,1681527213260.c44270c3412c22e2c805f466aae6ff56.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/c44270c3412c22e2c805f466aae6ff56/A/9b3a32d29fc74086b895f2308fd5a598 to hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/archive/data/default/TestAcidGuarantees/c44270c3412c22e2c805f466aae6ff56/A/9b3a32d29fc74086b895f2308fd5a598 2023-04-15 02:53:58,208 DEBUG [StoreCloser-TestAcidGuarantees,,1681527213260.c44270c3412c22e2c805f466aae6ff56.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/c44270c3412c22e2c805f466aae6ff56/A/6f6754556026426b882980c6173c34d5 to hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/archive/data/default/TestAcidGuarantees/c44270c3412c22e2c805f466aae6ff56/A/6f6754556026426b882980c6173c34d5 2023-04-15 02:53:58,209 DEBUG [StoreCloser-TestAcidGuarantees,,1681527213260.c44270c3412c22e2c805f466aae6ff56.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/c44270c3412c22e2c805f466aae6ff56/A/107ad8c17894439e9194631ffc9b3c4d to hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/archive/data/default/TestAcidGuarantees/c44270c3412c22e2c805f466aae6ff56/A/107ad8c17894439e9194631ffc9b3c4d 2023-04-15 02:53:58,210 DEBUG [StoreCloser-TestAcidGuarantees,,1681527213260.c44270c3412c22e2c805f466aae6ff56.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/c44270c3412c22e2c805f466aae6ff56/A/ff96f2210e8741199ea6a6a64e0596ad to hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/archive/data/default/TestAcidGuarantees/c44270c3412c22e2c805f466aae6ff56/A/ff96f2210e8741199ea6a6a64e0596ad 2023-04-15 02:53:58,212 DEBUG [StoreCloser-TestAcidGuarantees,,1681527213260.c44270c3412c22e2c805f466aae6ff56.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/c44270c3412c22e2c805f466aae6ff56/A/c14d9273a0a44d31ac175cdf798bf17b to hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/archive/data/default/TestAcidGuarantees/c44270c3412c22e2c805f466aae6ff56/A/c14d9273a0a44d31ac175cdf798bf17b 2023-04-15 02:53:58,213 DEBUG [StoreCloser-TestAcidGuarantees,,1681527213260.c44270c3412c22e2c805f466aae6ff56.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/c44270c3412c22e2c805f466aae6ff56/A/2573076332df46de97607613d26daddd to hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/archive/data/default/TestAcidGuarantees/c44270c3412c22e2c805f466aae6ff56/A/2573076332df46de97607613d26daddd 2023-04-15 02:53:58,214 DEBUG [StoreCloser-TestAcidGuarantees,,1681527213260.c44270c3412c22e2c805f466aae6ff56.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/c44270c3412c22e2c805f466aae6ff56/A/3a38a999ff8441b7b08fcc09067aca06 to hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/archive/data/default/TestAcidGuarantees/c44270c3412c22e2c805f466aae6ff56/A/3a38a999ff8441b7b08fcc09067aca06 2023-04-15 02:53:58,215 DEBUG [StoreCloser-TestAcidGuarantees,,1681527213260.c44270c3412c22e2c805f466aae6ff56.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/c44270c3412c22e2c805f466aae6ff56/A/58061441bf484cf5a57757d389a55706 to hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/archive/data/default/TestAcidGuarantees/c44270c3412c22e2c805f466aae6ff56/A/58061441bf484cf5a57757d389a55706 2023-04-15 02:53:58,216 DEBUG [StoreCloser-TestAcidGuarantees,,1681527213260.c44270c3412c22e2c805f466aae6ff56.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/c44270c3412c22e2c805f466aae6ff56/A/9adad4a9679d4598acc5a03552f1531e to hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/archive/data/default/TestAcidGuarantees/c44270c3412c22e2c805f466aae6ff56/A/9adad4a9679d4598acc5a03552f1531e 2023-04-15 02:53:58,217 DEBUG [StoreCloser-TestAcidGuarantees,,1681527213260.c44270c3412c22e2c805f466aae6ff56.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/c44270c3412c22e2c805f466aae6ff56/A/f5429821fb914e68ba7a97b8cf6ad6ad to hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/archive/data/default/TestAcidGuarantees/c44270c3412c22e2c805f466aae6ff56/A/f5429821fb914e68ba7a97b8cf6ad6ad 2023-04-15 02:53:58,218 DEBUG [StoreCloser-TestAcidGuarantees,,1681527213260.c44270c3412c22e2c805f466aae6ff56.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/c44270c3412c22e2c805f466aae6ff56/A/3e42386781194a9bab119952e28e6bba to hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/archive/data/default/TestAcidGuarantees/c44270c3412c22e2c805f466aae6ff56/A/3e42386781194a9bab119952e28e6bba 2023-04-15 02:53:58,242 DEBUG [StoreCloser-TestAcidGuarantees,,1681527213260.c44270c3412c22e2c805f466aae6ff56.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/c44270c3412c22e2c805f466aae6ff56/B/e9337a469b0144028bebf63d83f2cafb, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/c44270c3412c22e2c805f466aae6ff56/B/fe072ba56cc048e29f7a020e4e4f8eea, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/c44270c3412c22e2c805f466aae6ff56/B/02423a6e336a4e15a0d0c1700fc4e738, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/c44270c3412c22e2c805f466aae6ff56/B/b9105532a32d4897a770fbf662f69001, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/c44270c3412c22e2c805f466aae6ff56/B/c2cf376d42b84a2cb7ec7a3735e47c7d, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/c44270c3412c22e2c805f466aae6ff56/B/2b7f5f8d4c2641089789c42009ef4cef, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/c44270c3412c22e2c805f466aae6ff56/B/a51d0e04a11143799840143be80d3d30, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/c44270c3412c22e2c805f466aae6ff56/B/4189f9dcfd404ba88dba3d8442bbb7cb, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/c44270c3412c22e2c805f466aae6ff56/B/00e29523c81a4a4b8618ef27d74d4385, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/c44270c3412c22e2c805f466aae6ff56/B/5e3a6ea18fae44eaa9eeb08894d78035, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/c44270c3412c22e2c805f466aae6ff56/B/71f84bdaa00940eda909883ab8648894, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/c44270c3412c22e2c805f466aae6ff56/B/033565ea1a184d06b070bb8636f464a2, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/c44270c3412c22e2c805f466aae6ff56/B/e91a0eec44524aaea30d1409f1d7aac8, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/c44270c3412c22e2c805f466aae6ff56/B/ee3572e477764367b4360e7da2e9b6b9, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/c44270c3412c22e2c805f466aae6ff56/B/a517dd6eae5d4726a3b1947f94b7de28, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/c44270c3412c22e2c805f466aae6ff56/B/6e8954bf57a44d1598c74e296874d020, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/c44270c3412c22e2c805f466aae6ff56/B/575038b7953e422695ac1f08a68ff389, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/c44270c3412c22e2c805f466aae6ff56/B/26642d181a934868b18a7d75c282341b, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/c44270c3412c22e2c805f466aae6ff56/B/613efbbcf3014af29a33f60fe1db0ef2, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/c44270c3412c22e2c805f466aae6ff56/B/4c9e2f7065754ea4a844f9954a828880, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/c44270c3412c22e2c805f466aae6ff56/B/3fadaf364e634d6f808b345ecf287298, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/c44270c3412c22e2c805f466aae6ff56/B/fcd3cf354e5140479901c13ba09dae61, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/c44270c3412c22e2c805f466aae6ff56/B/7276d3ce7f4c4e78be690dabe67ac89c, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/c44270c3412c22e2c805f466aae6ff56/B/9afaa7d2a9be47d185532e68809bfe92, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/c44270c3412c22e2c805f466aae6ff56/B/00a3a9b0abc448ae9c1f19b38c7583ff, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/c44270c3412c22e2c805f466aae6ff56/B/7cec6ef3d3de46b88188034a7cdb0313, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/c44270c3412c22e2c805f466aae6ff56/B/4eb9fadc993646b894e80997d6e745ca, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/c44270c3412c22e2c805f466aae6ff56/B/0c1862f1233640a5810d6d90f85e4696] to archive 2023-04-15 02:53:58,243 DEBUG [StoreCloser-TestAcidGuarantees,,1681527213260.c44270c3412c22e2c805f466aae6ff56.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-15 02:53:58,244 DEBUG [StoreCloser-TestAcidGuarantees,,1681527213260.c44270c3412c22e2c805f466aae6ff56.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/c44270c3412c22e2c805f466aae6ff56/B/e9337a469b0144028bebf63d83f2cafb to hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/archive/data/default/TestAcidGuarantees/c44270c3412c22e2c805f466aae6ff56/B/e9337a469b0144028bebf63d83f2cafb 2023-04-15 02:53:58,245 DEBUG [StoreCloser-TestAcidGuarantees,,1681527213260.c44270c3412c22e2c805f466aae6ff56.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/c44270c3412c22e2c805f466aae6ff56/B/fe072ba56cc048e29f7a020e4e4f8eea to hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/archive/data/default/TestAcidGuarantees/c44270c3412c22e2c805f466aae6ff56/B/fe072ba56cc048e29f7a020e4e4f8eea 2023-04-15 02:53:58,247 DEBUG [StoreCloser-TestAcidGuarantees,,1681527213260.c44270c3412c22e2c805f466aae6ff56.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/c44270c3412c22e2c805f466aae6ff56/B/02423a6e336a4e15a0d0c1700fc4e738 to hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/archive/data/default/TestAcidGuarantees/c44270c3412c22e2c805f466aae6ff56/B/02423a6e336a4e15a0d0c1700fc4e738 2023-04-15 02:53:58,248 DEBUG [StoreCloser-TestAcidGuarantees,,1681527213260.c44270c3412c22e2c805f466aae6ff56.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/c44270c3412c22e2c805f466aae6ff56/B/b9105532a32d4897a770fbf662f69001 to hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/archive/data/default/TestAcidGuarantees/c44270c3412c22e2c805f466aae6ff56/B/b9105532a32d4897a770fbf662f69001 2023-04-15 02:53:58,250 DEBUG [StoreCloser-TestAcidGuarantees,,1681527213260.c44270c3412c22e2c805f466aae6ff56.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/c44270c3412c22e2c805f466aae6ff56/B/c2cf376d42b84a2cb7ec7a3735e47c7d to hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/archive/data/default/TestAcidGuarantees/c44270c3412c22e2c805f466aae6ff56/B/c2cf376d42b84a2cb7ec7a3735e47c7d 2023-04-15 02:53:58,251 DEBUG [StoreCloser-TestAcidGuarantees,,1681527213260.c44270c3412c22e2c805f466aae6ff56.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/c44270c3412c22e2c805f466aae6ff56/B/2b7f5f8d4c2641089789c42009ef4cef to hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/archive/data/default/TestAcidGuarantees/c44270c3412c22e2c805f466aae6ff56/B/2b7f5f8d4c2641089789c42009ef4cef 2023-04-15 02:53:58,252 DEBUG [StoreCloser-TestAcidGuarantees,,1681527213260.c44270c3412c22e2c805f466aae6ff56.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/c44270c3412c22e2c805f466aae6ff56/B/a51d0e04a11143799840143be80d3d30 to hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/archive/data/default/TestAcidGuarantees/c44270c3412c22e2c805f466aae6ff56/B/a51d0e04a11143799840143be80d3d30 2023-04-15 02:53:58,253 DEBUG [StoreCloser-TestAcidGuarantees,,1681527213260.c44270c3412c22e2c805f466aae6ff56.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/c44270c3412c22e2c805f466aae6ff56/B/4189f9dcfd404ba88dba3d8442bbb7cb to hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/archive/data/default/TestAcidGuarantees/c44270c3412c22e2c805f466aae6ff56/B/4189f9dcfd404ba88dba3d8442bbb7cb 2023-04-15 02:53:58,254 DEBUG [StoreCloser-TestAcidGuarantees,,1681527213260.c44270c3412c22e2c805f466aae6ff56.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/c44270c3412c22e2c805f466aae6ff56/B/00e29523c81a4a4b8618ef27d74d4385 to hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/archive/data/default/TestAcidGuarantees/c44270c3412c22e2c805f466aae6ff56/B/00e29523c81a4a4b8618ef27d74d4385 2023-04-15 02:53:58,255 DEBUG [StoreCloser-TestAcidGuarantees,,1681527213260.c44270c3412c22e2c805f466aae6ff56.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/c44270c3412c22e2c805f466aae6ff56/B/5e3a6ea18fae44eaa9eeb08894d78035 to hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/archive/data/default/TestAcidGuarantees/c44270c3412c22e2c805f466aae6ff56/B/5e3a6ea18fae44eaa9eeb08894d78035 2023-04-15 02:53:58,256 DEBUG [StoreCloser-TestAcidGuarantees,,1681527213260.c44270c3412c22e2c805f466aae6ff56.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/c44270c3412c22e2c805f466aae6ff56/B/71f84bdaa00940eda909883ab8648894 to hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/archive/data/default/TestAcidGuarantees/c44270c3412c22e2c805f466aae6ff56/B/71f84bdaa00940eda909883ab8648894 2023-04-15 02:53:58,257 DEBUG [StoreCloser-TestAcidGuarantees,,1681527213260.c44270c3412c22e2c805f466aae6ff56.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/c44270c3412c22e2c805f466aae6ff56/B/033565ea1a184d06b070bb8636f464a2 to hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/archive/data/default/TestAcidGuarantees/c44270c3412c22e2c805f466aae6ff56/B/033565ea1a184d06b070bb8636f464a2 2023-04-15 02:53:58,258 DEBUG [StoreCloser-TestAcidGuarantees,,1681527213260.c44270c3412c22e2c805f466aae6ff56.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/c44270c3412c22e2c805f466aae6ff56/B/e91a0eec44524aaea30d1409f1d7aac8 to hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/archive/data/default/TestAcidGuarantees/c44270c3412c22e2c805f466aae6ff56/B/e91a0eec44524aaea30d1409f1d7aac8 2023-04-15 02:53:58,259 DEBUG [StoreCloser-TestAcidGuarantees,,1681527213260.c44270c3412c22e2c805f466aae6ff56.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/c44270c3412c22e2c805f466aae6ff56/B/ee3572e477764367b4360e7da2e9b6b9 to hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/archive/data/default/TestAcidGuarantees/c44270c3412c22e2c805f466aae6ff56/B/ee3572e477764367b4360e7da2e9b6b9 2023-04-15 02:53:58,260 DEBUG [StoreCloser-TestAcidGuarantees,,1681527213260.c44270c3412c22e2c805f466aae6ff56.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/c44270c3412c22e2c805f466aae6ff56/B/a517dd6eae5d4726a3b1947f94b7de28 to hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/archive/data/default/TestAcidGuarantees/c44270c3412c22e2c805f466aae6ff56/B/a517dd6eae5d4726a3b1947f94b7de28 2023-04-15 02:53:58,261 DEBUG [StoreCloser-TestAcidGuarantees,,1681527213260.c44270c3412c22e2c805f466aae6ff56.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/c44270c3412c22e2c805f466aae6ff56/B/6e8954bf57a44d1598c74e296874d020 to hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/archive/data/default/TestAcidGuarantees/c44270c3412c22e2c805f466aae6ff56/B/6e8954bf57a44d1598c74e296874d020 2023-04-15 02:53:58,262 DEBUG [StoreCloser-TestAcidGuarantees,,1681527213260.c44270c3412c22e2c805f466aae6ff56.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/c44270c3412c22e2c805f466aae6ff56/B/575038b7953e422695ac1f08a68ff389 to hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/archive/data/default/TestAcidGuarantees/c44270c3412c22e2c805f466aae6ff56/B/575038b7953e422695ac1f08a68ff389 2023-04-15 02:53:58,263 DEBUG [StoreCloser-TestAcidGuarantees,,1681527213260.c44270c3412c22e2c805f466aae6ff56.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/c44270c3412c22e2c805f466aae6ff56/B/26642d181a934868b18a7d75c282341b to hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/archive/data/default/TestAcidGuarantees/c44270c3412c22e2c805f466aae6ff56/B/26642d181a934868b18a7d75c282341b 2023-04-15 02:53:58,264 DEBUG [StoreCloser-TestAcidGuarantees,,1681527213260.c44270c3412c22e2c805f466aae6ff56.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/c44270c3412c22e2c805f466aae6ff56/B/613efbbcf3014af29a33f60fe1db0ef2 to hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/archive/data/default/TestAcidGuarantees/c44270c3412c22e2c805f466aae6ff56/B/613efbbcf3014af29a33f60fe1db0ef2 2023-04-15 02:53:58,265 DEBUG [StoreCloser-TestAcidGuarantees,,1681527213260.c44270c3412c22e2c805f466aae6ff56.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/c44270c3412c22e2c805f466aae6ff56/B/4c9e2f7065754ea4a844f9954a828880 to hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/archive/data/default/TestAcidGuarantees/c44270c3412c22e2c805f466aae6ff56/B/4c9e2f7065754ea4a844f9954a828880 2023-04-15 02:53:58,266 DEBUG [StoreCloser-TestAcidGuarantees,,1681527213260.c44270c3412c22e2c805f466aae6ff56.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/c44270c3412c22e2c805f466aae6ff56/B/3fadaf364e634d6f808b345ecf287298 to hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/archive/data/default/TestAcidGuarantees/c44270c3412c22e2c805f466aae6ff56/B/3fadaf364e634d6f808b345ecf287298 2023-04-15 02:53:58,267 DEBUG [StoreCloser-TestAcidGuarantees,,1681527213260.c44270c3412c22e2c805f466aae6ff56.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/c44270c3412c22e2c805f466aae6ff56/B/fcd3cf354e5140479901c13ba09dae61 to hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/archive/data/default/TestAcidGuarantees/c44270c3412c22e2c805f466aae6ff56/B/fcd3cf354e5140479901c13ba09dae61 2023-04-15 02:53:58,268 DEBUG [StoreCloser-TestAcidGuarantees,,1681527213260.c44270c3412c22e2c805f466aae6ff56.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/c44270c3412c22e2c805f466aae6ff56/B/7276d3ce7f4c4e78be690dabe67ac89c to hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/archive/data/default/TestAcidGuarantees/c44270c3412c22e2c805f466aae6ff56/B/7276d3ce7f4c4e78be690dabe67ac89c 2023-04-15 02:53:58,269 DEBUG [StoreCloser-TestAcidGuarantees,,1681527213260.c44270c3412c22e2c805f466aae6ff56.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/c44270c3412c22e2c805f466aae6ff56/B/9afaa7d2a9be47d185532e68809bfe92 to hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/archive/data/default/TestAcidGuarantees/c44270c3412c22e2c805f466aae6ff56/B/9afaa7d2a9be47d185532e68809bfe92 2023-04-15 02:53:58,270 DEBUG [StoreCloser-TestAcidGuarantees,,1681527213260.c44270c3412c22e2c805f466aae6ff56.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/c44270c3412c22e2c805f466aae6ff56/B/00a3a9b0abc448ae9c1f19b38c7583ff to hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/archive/data/default/TestAcidGuarantees/c44270c3412c22e2c805f466aae6ff56/B/00a3a9b0abc448ae9c1f19b38c7583ff 2023-04-15 02:53:58,271 DEBUG [StoreCloser-TestAcidGuarantees,,1681527213260.c44270c3412c22e2c805f466aae6ff56.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/c44270c3412c22e2c805f466aae6ff56/B/7cec6ef3d3de46b88188034a7cdb0313 to hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/archive/data/default/TestAcidGuarantees/c44270c3412c22e2c805f466aae6ff56/B/7cec6ef3d3de46b88188034a7cdb0313 2023-04-15 02:53:58,272 DEBUG [StoreCloser-TestAcidGuarantees,,1681527213260.c44270c3412c22e2c805f466aae6ff56.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/c44270c3412c22e2c805f466aae6ff56/B/4eb9fadc993646b894e80997d6e745ca to hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/archive/data/default/TestAcidGuarantees/c44270c3412c22e2c805f466aae6ff56/B/4eb9fadc993646b894e80997d6e745ca 2023-04-15 02:53:58,273 DEBUG [StoreCloser-TestAcidGuarantees,,1681527213260.c44270c3412c22e2c805f466aae6ff56.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/c44270c3412c22e2c805f466aae6ff56/B/0c1862f1233640a5810d6d90f85e4696 to hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/archive/data/default/TestAcidGuarantees/c44270c3412c22e2c805f466aae6ff56/B/0c1862f1233640a5810d6d90f85e4696 2023-04-15 02:53:58,276 DEBUG [StoreCloser-TestAcidGuarantees,,1681527213260.c44270c3412c22e2c805f466aae6ff56.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/c44270c3412c22e2c805f466aae6ff56/C/081cf4a0d52047808a158eecff6304fe, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/c44270c3412c22e2c805f466aae6ff56/C/767f544ff04444b4b1e74da69ea1dc48, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/c44270c3412c22e2c805f466aae6ff56/C/d3fb7497c5d54289bef7742bb9d41db6, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/c44270c3412c22e2c805f466aae6ff56/C/0789443aa76c4c5a845f56519c2f3c41, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/c44270c3412c22e2c805f466aae6ff56/C/fe70e4bee82f4087938046976ab556b6, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/c44270c3412c22e2c805f466aae6ff56/C/f5e6234a44594733a559527c3e6a1f63, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/c44270c3412c22e2c805f466aae6ff56/C/bd10531039a0472b8dd505a7b846eabc, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/c44270c3412c22e2c805f466aae6ff56/C/1adba151c2d2465e8f2ae5359aad1ba5, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/c44270c3412c22e2c805f466aae6ff56/C/c3ee48d87cbb43b6b0cc49b0a53b533f, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/c44270c3412c22e2c805f466aae6ff56/C/4f01c5f98a3d4c2f8c452865f2ab23f0, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/c44270c3412c22e2c805f466aae6ff56/C/845decfbcaff4e56a8eb1e0a5db76d11, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/c44270c3412c22e2c805f466aae6ff56/C/4cb2f6c7cea1426cb832a4e7203a86bc, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/c44270c3412c22e2c805f466aae6ff56/C/1fb73c4dcf674ed099e6e97c11f99372, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/c44270c3412c22e2c805f466aae6ff56/C/ff9445b047e14cd4ace2d569eb140ab6, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/c44270c3412c22e2c805f466aae6ff56/C/935006ea70d34e00b33b6556dfd065fd, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/c44270c3412c22e2c805f466aae6ff56/C/c4954ab54de54450a33fc1030524f6ed, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/c44270c3412c22e2c805f466aae6ff56/C/d10899e00fe64a74810f1c8fe7034cc5, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/c44270c3412c22e2c805f466aae6ff56/C/bc739284553c434bbe2f22067bfb697a, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/c44270c3412c22e2c805f466aae6ff56/C/d189626d0b484141936ae0d230907be8, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/c44270c3412c22e2c805f466aae6ff56/C/c706a717a77e466ab0d6d968caccf6ce, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/c44270c3412c22e2c805f466aae6ff56/C/42d7ae1915bf4283935d2781d54de6d7, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/c44270c3412c22e2c805f466aae6ff56/C/3de6cdf4cc5f47ff9def540e573bb52d, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/c44270c3412c22e2c805f466aae6ff56/C/fc51d87e600940bda48e00d50d6afe40, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/c44270c3412c22e2c805f466aae6ff56/C/f6f90e98a27f4485ba53d503154f7322, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/c44270c3412c22e2c805f466aae6ff56/C/4f1c63d6ddf542c6bd87f7fc14334ff0, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/c44270c3412c22e2c805f466aae6ff56/C/37ee1c600b8f4fe794a27f1f1edfa0fd, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/c44270c3412c22e2c805f466aae6ff56/C/81b3e7d008db4b72a7cea39386d1e7a7, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/c44270c3412c22e2c805f466aae6ff56/C/0f371f625e8a4376b07e78dc48ad8518] to archive 2023-04-15 02:53:58,277 DEBUG [StoreCloser-TestAcidGuarantees,,1681527213260.c44270c3412c22e2c805f466aae6ff56.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-15 02:53:58,279 DEBUG [StoreCloser-TestAcidGuarantees,,1681527213260.c44270c3412c22e2c805f466aae6ff56.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/c44270c3412c22e2c805f466aae6ff56/C/081cf4a0d52047808a158eecff6304fe to hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/archive/data/default/TestAcidGuarantees/c44270c3412c22e2c805f466aae6ff56/C/081cf4a0d52047808a158eecff6304fe 2023-04-15 02:53:58,280 DEBUG [StoreCloser-TestAcidGuarantees,,1681527213260.c44270c3412c22e2c805f466aae6ff56.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/c44270c3412c22e2c805f466aae6ff56/C/767f544ff04444b4b1e74da69ea1dc48 to hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/archive/data/default/TestAcidGuarantees/c44270c3412c22e2c805f466aae6ff56/C/767f544ff04444b4b1e74da69ea1dc48 2023-04-15 02:53:58,281 DEBUG [StoreCloser-TestAcidGuarantees,,1681527213260.c44270c3412c22e2c805f466aae6ff56.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/c44270c3412c22e2c805f466aae6ff56/C/d3fb7497c5d54289bef7742bb9d41db6 to hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/archive/data/default/TestAcidGuarantees/c44270c3412c22e2c805f466aae6ff56/C/d3fb7497c5d54289bef7742bb9d41db6 2023-04-15 02:53:58,282 DEBUG [StoreCloser-TestAcidGuarantees,,1681527213260.c44270c3412c22e2c805f466aae6ff56.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/c44270c3412c22e2c805f466aae6ff56/C/0789443aa76c4c5a845f56519c2f3c41 to hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/archive/data/default/TestAcidGuarantees/c44270c3412c22e2c805f466aae6ff56/C/0789443aa76c4c5a845f56519c2f3c41 2023-04-15 02:53:58,283 DEBUG [StoreCloser-TestAcidGuarantees,,1681527213260.c44270c3412c22e2c805f466aae6ff56.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/c44270c3412c22e2c805f466aae6ff56/C/fe70e4bee82f4087938046976ab556b6 to hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/archive/data/default/TestAcidGuarantees/c44270c3412c22e2c805f466aae6ff56/C/fe70e4bee82f4087938046976ab556b6 2023-04-15 02:53:58,284 DEBUG [StoreCloser-TestAcidGuarantees,,1681527213260.c44270c3412c22e2c805f466aae6ff56.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/c44270c3412c22e2c805f466aae6ff56/C/f5e6234a44594733a559527c3e6a1f63 to hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/archive/data/default/TestAcidGuarantees/c44270c3412c22e2c805f466aae6ff56/C/f5e6234a44594733a559527c3e6a1f63 2023-04-15 02:53:58,285 DEBUG [StoreCloser-TestAcidGuarantees,,1681527213260.c44270c3412c22e2c805f466aae6ff56.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/c44270c3412c22e2c805f466aae6ff56/C/bd10531039a0472b8dd505a7b846eabc to hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/archive/data/default/TestAcidGuarantees/c44270c3412c22e2c805f466aae6ff56/C/bd10531039a0472b8dd505a7b846eabc 2023-04-15 02:53:58,286 DEBUG [StoreCloser-TestAcidGuarantees,,1681527213260.c44270c3412c22e2c805f466aae6ff56.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/c44270c3412c22e2c805f466aae6ff56/C/1adba151c2d2465e8f2ae5359aad1ba5 to hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/archive/data/default/TestAcidGuarantees/c44270c3412c22e2c805f466aae6ff56/C/1adba151c2d2465e8f2ae5359aad1ba5 2023-04-15 02:53:58,288 DEBUG [StoreCloser-TestAcidGuarantees,,1681527213260.c44270c3412c22e2c805f466aae6ff56.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/c44270c3412c22e2c805f466aae6ff56/C/c3ee48d87cbb43b6b0cc49b0a53b533f to hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/archive/data/default/TestAcidGuarantees/c44270c3412c22e2c805f466aae6ff56/C/c3ee48d87cbb43b6b0cc49b0a53b533f 2023-04-15 02:53:58,289 DEBUG [StoreCloser-TestAcidGuarantees,,1681527213260.c44270c3412c22e2c805f466aae6ff56.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/c44270c3412c22e2c805f466aae6ff56/C/4f01c5f98a3d4c2f8c452865f2ab23f0 to hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/archive/data/default/TestAcidGuarantees/c44270c3412c22e2c805f466aae6ff56/C/4f01c5f98a3d4c2f8c452865f2ab23f0 2023-04-15 02:53:58,289 DEBUG [StoreCloser-TestAcidGuarantees,,1681527213260.c44270c3412c22e2c805f466aae6ff56.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/c44270c3412c22e2c805f466aae6ff56/C/845decfbcaff4e56a8eb1e0a5db76d11 to hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/archive/data/default/TestAcidGuarantees/c44270c3412c22e2c805f466aae6ff56/C/845decfbcaff4e56a8eb1e0a5db76d11 2023-04-15 02:53:58,290 DEBUG [StoreCloser-TestAcidGuarantees,,1681527213260.c44270c3412c22e2c805f466aae6ff56.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/c44270c3412c22e2c805f466aae6ff56/C/4cb2f6c7cea1426cb832a4e7203a86bc to hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/archive/data/default/TestAcidGuarantees/c44270c3412c22e2c805f466aae6ff56/C/4cb2f6c7cea1426cb832a4e7203a86bc 2023-04-15 02:53:58,291 DEBUG [StoreCloser-TestAcidGuarantees,,1681527213260.c44270c3412c22e2c805f466aae6ff56.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/c44270c3412c22e2c805f466aae6ff56/C/1fb73c4dcf674ed099e6e97c11f99372 to hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/archive/data/default/TestAcidGuarantees/c44270c3412c22e2c805f466aae6ff56/C/1fb73c4dcf674ed099e6e97c11f99372 2023-04-15 02:53:58,292 DEBUG [StoreCloser-TestAcidGuarantees,,1681527213260.c44270c3412c22e2c805f466aae6ff56.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/c44270c3412c22e2c805f466aae6ff56/C/ff9445b047e14cd4ace2d569eb140ab6 to hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/archive/data/default/TestAcidGuarantees/c44270c3412c22e2c805f466aae6ff56/C/ff9445b047e14cd4ace2d569eb140ab6 2023-04-15 02:53:58,293 DEBUG [StoreCloser-TestAcidGuarantees,,1681527213260.c44270c3412c22e2c805f466aae6ff56.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/c44270c3412c22e2c805f466aae6ff56/C/935006ea70d34e00b33b6556dfd065fd to hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/archive/data/default/TestAcidGuarantees/c44270c3412c22e2c805f466aae6ff56/C/935006ea70d34e00b33b6556dfd065fd 2023-04-15 02:53:58,294 DEBUG [StoreCloser-TestAcidGuarantees,,1681527213260.c44270c3412c22e2c805f466aae6ff56.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/c44270c3412c22e2c805f466aae6ff56/C/c4954ab54de54450a33fc1030524f6ed to hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/archive/data/default/TestAcidGuarantees/c44270c3412c22e2c805f466aae6ff56/C/c4954ab54de54450a33fc1030524f6ed 2023-04-15 02:53:58,295 DEBUG [StoreCloser-TestAcidGuarantees,,1681527213260.c44270c3412c22e2c805f466aae6ff56.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/c44270c3412c22e2c805f466aae6ff56/C/d10899e00fe64a74810f1c8fe7034cc5 to hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/archive/data/default/TestAcidGuarantees/c44270c3412c22e2c805f466aae6ff56/C/d10899e00fe64a74810f1c8fe7034cc5 2023-04-15 02:53:58,296 DEBUG [StoreCloser-TestAcidGuarantees,,1681527213260.c44270c3412c22e2c805f466aae6ff56.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/c44270c3412c22e2c805f466aae6ff56/C/bc739284553c434bbe2f22067bfb697a to hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/archive/data/default/TestAcidGuarantees/c44270c3412c22e2c805f466aae6ff56/C/bc739284553c434bbe2f22067bfb697a 2023-04-15 02:53:58,297 DEBUG [StoreCloser-TestAcidGuarantees,,1681527213260.c44270c3412c22e2c805f466aae6ff56.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/c44270c3412c22e2c805f466aae6ff56/C/d189626d0b484141936ae0d230907be8 to hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/archive/data/default/TestAcidGuarantees/c44270c3412c22e2c805f466aae6ff56/C/d189626d0b484141936ae0d230907be8 2023-04-15 02:53:58,298 DEBUG [StoreCloser-TestAcidGuarantees,,1681527213260.c44270c3412c22e2c805f466aae6ff56.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/c44270c3412c22e2c805f466aae6ff56/C/c706a717a77e466ab0d6d968caccf6ce to hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/archive/data/default/TestAcidGuarantees/c44270c3412c22e2c805f466aae6ff56/C/c706a717a77e466ab0d6d968caccf6ce 2023-04-15 02:53:58,298 DEBUG [StoreCloser-TestAcidGuarantees,,1681527213260.c44270c3412c22e2c805f466aae6ff56.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/c44270c3412c22e2c805f466aae6ff56/C/42d7ae1915bf4283935d2781d54de6d7 to hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/archive/data/default/TestAcidGuarantees/c44270c3412c22e2c805f466aae6ff56/C/42d7ae1915bf4283935d2781d54de6d7 2023-04-15 02:53:58,299 DEBUG [StoreCloser-TestAcidGuarantees,,1681527213260.c44270c3412c22e2c805f466aae6ff56.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/c44270c3412c22e2c805f466aae6ff56/C/3de6cdf4cc5f47ff9def540e573bb52d to hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/archive/data/default/TestAcidGuarantees/c44270c3412c22e2c805f466aae6ff56/C/3de6cdf4cc5f47ff9def540e573bb52d 2023-04-15 02:53:58,300 DEBUG [StoreCloser-TestAcidGuarantees,,1681527213260.c44270c3412c22e2c805f466aae6ff56.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/c44270c3412c22e2c805f466aae6ff56/C/fc51d87e600940bda48e00d50d6afe40 to hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/archive/data/default/TestAcidGuarantees/c44270c3412c22e2c805f466aae6ff56/C/fc51d87e600940bda48e00d50d6afe40 2023-04-15 02:53:58,301 DEBUG [StoreCloser-TestAcidGuarantees,,1681527213260.c44270c3412c22e2c805f466aae6ff56.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/c44270c3412c22e2c805f466aae6ff56/C/f6f90e98a27f4485ba53d503154f7322 to hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/archive/data/default/TestAcidGuarantees/c44270c3412c22e2c805f466aae6ff56/C/f6f90e98a27f4485ba53d503154f7322 2023-04-15 02:53:58,303 DEBUG [StoreCloser-TestAcidGuarantees,,1681527213260.c44270c3412c22e2c805f466aae6ff56.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/c44270c3412c22e2c805f466aae6ff56/C/4f1c63d6ddf542c6bd87f7fc14334ff0 to hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/archive/data/default/TestAcidGuarantees/c44270c3412c22e2c805f466aae6ff56/C/4f1c63d6ddf542c6bd87f7fc14334ff0 2023-04-15 02:53:58,304 DEBUG [StoreCloser-TestAcidGuarantees,,1681527213260.c44270c3412c22e2c805f466aae6ff56.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/c44270c3412c22e2c805f466aae6ff56/C/37ee1c600b8f4fe794a27f1f1edfa0fd to hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/archive/data/default/TestAcidGuarantees/c44270c3412c22e2c805f466aae6ff56/C/37ee1c600b8f4fe794a27f1f1edfa0fd 2023-04-15 02:53:58,305 DEBUG [StoreCloser-TestAcidGuarantees,,1681527213260.c44270c3412c22e2c805f466aae6ff56.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/c44270c3412c22e2c805f466aae6ff56/C/81b3e7d008db4b72a7cea39386d1e7a7 to hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/archive/data/default/TestAcidGuarantees/c44270c3412c22e2c805f466aae6ff56/C/81b3e7d008db4b72a7cea39386d1e7a7 2023-04-15 02:53:58,306 DEBUG [StoreCloser-TestAcidGuarantees,,1681527213260.c44270c3412c22e2c805f466aae6ff56.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/c44270c3412c22e2c805f466aae6ff56/C/0f371f625e8a4376b07e78dc48ad8518 to hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/archive/data/default/TestAcidGuarantees/c44270c3412c22e2c805f466aae6ff56/C/0f371f625e8a4376b07e78dc48ad8518 2023-04-15 02:53:58,312 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/c44270c3412c22e2c805f466aae6ff56/recovered.edits/563.seqid, newMaxSeqId=563, maxSeqId=1 2023-04-15 02:53:58,314 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1681527213260.c44270c3412c22e2c805f466aae6ff56. 2023-04-15 02:53:58,314 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1558): Region close journal for c44270c3412c22e2c805f466aae6ff56: 2023-04-15 02:53:58,316 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] handler.UnassignRegionHandler(147): Closed c44270c3412c22e2c805f466aae6ff56 2023-04-15 02:53:58,316 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=13 updating hbase:meta row=c44270c3412c22e2c805f466aae6ff56, regionState=CLOSED 2023-04-15 02:53:58,317 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1681527213260.c44270c3412c22e2c805f466aae6ff56.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1681527238316"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1681527238316"}]},"ts":"1681527238316"} 2023-04-15 02:53:58,319 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=14, resume processing ppid=13 2023-04-15 02:53:58,319 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=14, ppid=13, state=SUCCESS; CloseRegionProcedure c44270c3412c22e2c805f466aae6ff56, server=jenkins-hbase20.apache.org,34681,1681527209133 in 2.3900 sec 2023-04-15 02:53:58,321 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=13, resume processing ppid=12 2023-04-15 02:53:58,321 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=13, ppid=12, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=c44270c3412c22e2c805f466aae6ff56, UNASSIGN in 2.3940 sec 2023-04-15 02:53:58,322 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1681527238322"}]},"ts":"1681527238322"} 2023-04-15 02:53:58,323 INFO [PEWorker-1] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=DISABLED in hbase:meta 2023-04-15 02:53:58,341 INFO [PEWorker-1] procedure.DisableTableProcedure(305): Set TestAcidGuarantees to state=DISABLED 2023-04-15 02:53:58,343 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=12, state=SUCCESS; DisableTableProcedure table=TestAcidGuarantees in 2.4280 sec 2023-04-15 02:54:00,031 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(1227): Checking to see if procedure is done pid=12 2023-04-15 02:54:00,032 INFO [Listener at localhost.localdomain/33693] client.HBaseAdmin$TableFuture(3541): Operation: DISABLE, Table Name: default:TestAcidGuarantees, procId: 12 completed 2023-04-15 02:54:00,036 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.HMaster$5(2228): Client=jenkins//148.251.75.209 delete TestAcidGuarantees 2023-04-15 02:54:00,042 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] procedure2.ProcedureExecutor(1029): Stored pid=15, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION; DeleteTableProcedure table=TestAcidGuarantees 2023-04-15 02:54:00,044 DEBUG [PEWorker-3] procedure.DeleteTableProcedure(101): Waiting for RIT for pid=15, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-15 02:54:00,045 DEBUG [PEWorker-3] procedure.DeleteTableProcedure(113): Deleting regions from filesystem for pid=15, state=RUNNABLE:DELETE_TABLE_CLEAR_FS_LAYOUT, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-15 02:54:00,046 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(1227): Checking to see if procedure is done pid=15 2023-04-15 02:54:00,050 DEBUG [HFileArchiver-3] backup.HFileArchiver(131): ARCHIVING hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/.tmp/data/default/TestAcidGuarantees/c44270c3412c22e2c805f466aae6ff56 2023-04-15 02:54:00,054 DEBUG [HFileArchiver-3] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/.tmp/data/default/TestAcidGuarantees/c44270c3412c22e2c805f466aae6ff56/A, FileablePath, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/.tmp/data/default/TestAcidGuarantees/c44270c3412c22e2c805f466aae6ff56/B, FileablePath, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/.tmp/data/default/TestAcidGuarantees/c44270c3412c22e2c805f466aae6ff56/C, FileablePath, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/.tmp/data/default/TestAcidGuarantees/c44270c3412c22e2c805f466aae6ff56/recovered.edits] 2023-04-15 02:54:00,057 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/.tmp/data/default/TestAcidGuarantees/c44270c3412c22e2c805f466aae6ff56/A/13ae9b1c4aac440fa3bf08f114450512 to hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/archive/data/default/TestAcidGuarantees/c44270c3412c22e2c805f466aae6ff56/A/13ae9b1c4aac440fa3bf08f114450512 2023-04-15 02:54:00,059 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/.tmp/data/default/TestAcidGuarantees/c44270c3412c22e2c805f466aae6ff56/A/2aaabf9165124956aa0d996a968ab13c to hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/archive/data/default/TestAcidGuarantees/c44270c3412c22e2c805f466aae6ff56/A/2aaabf9165124956aa0d996a968ab13c 2023-04-15 02:54:00,060 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/.tmp/data/default/TestAcidGuarantees/c44270c3412c22e2c805f466aae6ff56/A/4256873ddad14121a573f9103e7e2518 to hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/archive/data/default/TestAcidGuarantees/c44270c3412c22e2c805f466aae6ff56/A/4256873ddad14121a573f9103e7e2518 2023-04-15 02:54:00,061 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/.tmp/data/default/TestAcidGuarantees/c44270c3412c22e2c805f466aae6ff56/A/47d9637ccbe249a8a1c045372007b248 to hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/archive/data/default/TestAcidGuarantees/c44270c3412c22e2c805f466aae6ff56/A/47d9637ccbe249a8a1c045372007b248 2023-04-15 02:54:00,063 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/.tmp/data/default/TestAcidGuarantees/c44270c3412c22e2c805f466aae6ff56/A/67683853907140e0b4f8bebc8daa4dba to hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/archive/data/default/TestAcidGuarantees/c44270c3412c22e2c805f466aae6ff56/A/67683853907140e0b4f8bebc8daa4dba 2023-04-15 02:54:00,064 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/.tmp/data/default/TestAcidGuarantees/c44270c3412c22e2c805f466aae6ff56/A/7c3478a288494390bd0d3ef65f83a721 to hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/archive/data/default/TestAcidGuarantees/c44270c3412c22e2c805f466aae6ff56/A/7c3478a288494390bd0d3ef65f83a721 2023-04-15 02:54:00,068 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/.tmp/data/default/TestAcidGuarantees/c44270c3412c22e2c805f466aae6ff56/B/4a91b88ad17043e3abb471b5c4bd38f6 to hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/archive/data/default/TestAcidGuarantees/c44270c3412c22e2c805f466aae6ff56/B/4a91b88ad17043e3abb471b5c4bd38f6 2023-04-15 02:54:00,069 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/.tmp/data/default/TestAcidGuarantees/c44270c3412c22e2c805f466aae6ff56/B/9969523cf245497292e9adb1cd9fb873 to hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/archive/data/default/TestAcidGuarantees/c44270c3412c22e2c805f466aae6ff56/B/9969523cf245497292e9adb1cd9fb873 2023-04-15 02:54:00,070 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/.tmp/data/default/TestAcidGuarantees/c44270c3412c22e2c805f466aae6ff56/B/d1d472710d0c479aaa793ef456adc49f to hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/archive/data/default/TestAcidGuarantees/c44270c3412c22e2c805f466aae6ff56/B/d1d472710d0c479aaa793ef456adc49f 2023-04-15 02:54:00,072 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/.tmp/data/default/TestAcidGuarantees/c44270c3412c22e2c805f466aae6ff56/B/eb8fa00880054cf4b77f18772385523e to hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/archive/data/default/TestAcidGuarantees/c44270c3412c22e2c805f466aae6ff56/B/eb8fa00880054cf4b77f18772385523e 2023-04-15 02:54:00,073 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/.tmp/data/default/TestAcidGuarantees/c44270c3412c22e2c805f466aae6ff56/B/ed854dd66ac04f8fbefcf4e10188adb1 to hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/archive/data/default/TestAcidGuarantees/c44270c3412c22e2c805f466aae6ff56/B/ed854dd66ac04f8fbefcf4e10188adb1 2023-04-15 02:54:00,074 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/.tmp/data/default/TestAcidGuarantees/c44270c3412c22e2c805f466aae6ff56/B/f27a2ce0602a4e448c49475f2b96bacd to hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/archive/data/default/TestAcidGuarantees/c44270c3412c22e2c805f466aae6ff56/B/f27a2ce0602a4e448c49475f2b96bacd 2023-04-15 02:54:00,076 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/.tmp/data/default/TestAcidGuarantees/c44270c3412c22e2c805f466aae6ff56/C/7b936adb3bbb40e4ad1e9ee9845861ae to hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/archive/data/default/TestAcidGuarantees/c44270c3412c22e2c805f466aae6ff56/C/7b936adb3bbb40e4ad1e9ee9845861ae 2023-04-15 02:54:00,077 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/.tmp/data/default/TestAcidGuarantees/c44270c3412c22e2c805f466aae6ff56/C/b4c740841db14dbe87104caa2ce01425 to hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/archive/data/default/TestAcidGuarantees/c44270c3412c22e2c805f466aae6ff56/C/b4c740841db14dbe87104caa2ce01425 2023-04-15 02:54:00,078 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/.tmp/data/default/TestAcidGuarantees/c44270c3412c22e2c805f466aae6ff56/C/b605227d5ffe4ef8b2da2e8de095324e to hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/archive/data/default/TestAcidGuarantees/c44270c3412c22e2c805f466aae6ff56/C/b605227d5ffe4ef8b2da2e8de095324e 2023-04-15 02:54:00,079 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/.tmp/data/default/TestAcidGuarantees/c44270c3412c22e2c805f466aae6ff56/C/f41b5d2d34cc4d95ac9265fdc6c535f6 to hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/archive/data/default/TestAcidGuarantees/c44270c3412c22e2c805f466aae6ff56/C/f41b5d2d34cc4d95ac9265fdc6c535f6 2023-04-15 02:54:00,080 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/.tmp/data/default/TestAcidGuarantees/c44270c3412c22e2c805f466aae6ff56/C/f9ce79cd57ec4075a6436a49d4b44314 to hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/archive/data/default/TestAcidGuarantees/c44270c3412c22e2c805f466aae6ff56/C/f9ce79cd57ec4075a6436a49d4b44314 2023-04-15 02:54:00,081 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/.tmp/data/default/TestAcidGuarantees/c44270c3412c22e2c805f466aae6ff56/C/fbbfb964ab7744078ba460854e9c539e to hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/archive/data/default/TestAcidGuarantees/c44270c3412c22e2c805f466aae6ff56/C/fbbfb964ab7744078ba460854e9c539e 2023-04-15 02:54:00,083 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/.tmp/data/default/TestAcidGuarantees/c44270c3412c22e2c805f466aae6ff56/recovered.edits/563.seqid to hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/archive/data/default/TestAcidGuarantees/c44270c3412c22e2c805f466aae6ff56/recovered.edits/563.seqid 2023-04-15 02:54:00,084 DEBUG [HFileArchiver-3] backup.HFileArchiver(596): Deleted hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/.tmp/data/default/TestAcidGuarantees/c44270c3412c22e2c805f466aae6ff56 2023-04-15 02:54:00,084 DEBUG [PEWorker-3] procedure.DeleteTableProcedure(328): Archived TestAcidGuarantees regions 2023-04-15 02:54:00,086 DEBUG [PEWorker-3] procedure.DeleteTableProcedure(118): Deleting regions from META for pid=15, state=RUNNABLE:DELETE_TABLE_REMOVE_FROM_META, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-15 02:54:00,102 WARN [PEWorker-3] procedure.DeleteTableProcedure(384): Deleting some vestigial 1 rows of TestAcidGuarantees from hbase:meta 2023-04-15 02:54:00,135 DEBUG [PEWorker-3] procedure.DeleteTableProcedure(421): Removing 'TestAcidGuarantees' descriptor. 2023-04-15 02:54:00,136 DEBUG [PEWorker-3] procedure.DeleteTableProcedure(124): Deleting assignment state for pid=15, state=RUNNABLE:DELETE_TABLE_UNASSIGN_REGIONS, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-15 02:54:00,136 DEBUG [PEWorker-3] procedure.DeleteTableProcedure(411): Removing 'TestAcidGuarantees' from region states. 2023-04-15 02:54:00,136 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"TestAcidGuarantees,,1681527213260.c44270c3412c22e2c805f466aae6ff56.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1681527240136"}]},"ts":"9223372036854775807"} 2023-04-15 02:54:00,138 INFO [PEWorker-3] hbase.MetaTableAccessor(1788): Deleted 1 regions from META 2023-04-15 02:54:00,139 DEBUG [PEWorker-3] hbase.MetaTableAccessor(1789): Deleted regions: [{ENCODED => c44270c3412c22e2c805f466aae6ff56, NAME => 'TestAcidGuarantees,,1681527213260.c44270c3412c22e2c805f466aae6ff56.', STARTKEY => '', ENDKEY => ''}] 2023-04-15 02:54:00,139 DEBUG [PEWorker-3] procedure.DeleteTableProcedure(415): Marking 'TestAcidGuarantees' as deleted. 2023-04-15 02:54:00,139 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":0,"tag":[],"timestamp":"1681527240139"}]},"ts":"9223372036854775807"} 2023-04-15 02:54:00,141 INFO [PEWorker-3] hbase.MetaTableAccessor(1658): Deleted table TestAcidGuarantees state from META 2023-04-15 02:54:00,142 DEBUG [PEWorker-3] procedure.DeleteTableProcedure(130): Finished pid=15, state=RUNNABLE:DELETE_TABLE_POST_OPERATION, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-15 02:54:00,143 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=15, state=SUCCESS; DeleteTableProcedure table=TestAcidGuarantees in 105 msec 2023-04-15 02:54:00,148 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(1227): Checking to see if procedure is done pid=15 2023-04-15 02:54:00,148 INFO [Listener at localhost.localdomain/33693] client.HBaseAdmin$TableFuture(3541): Operation: DELETE, Table Name: default:TestAcidGuarantees, procId: 15 completed 2023-04-15 02:54:00,168 INFO [Listener at localhost.localdomain/33693] hbase.ResourceChecker(175): after: TestAcidGuaranteesWithEagerPolicy#testMixedAtomicity Thread=234 (was 207) Potentially hanging thread: RPCClient-NioEventLoopGroup-4-5 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-4-11 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-4-2 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:62) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:883) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: hconnection-0x48646271-shared-pool-12 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-4 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-6 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-4-6 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-1 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-5 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: (jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:458) java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362) java.util.concurrent.SynchronousQueue.take(SynchronousQueue.java:924) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS-EventLoopGroup-1-3 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:209) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:202) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWaitNoTimerChange(EpollEventLoop.java:306) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:363) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RSProcedureDispatcher-pool-2 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-4-10 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-4-8 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-0 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-7 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-4-12 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: HFileArchiver-2 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_-2053686367_15 at /127.0.0.1:44422 [Waiting for operation #3] sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:246) java.io.BufferedInputStream.read(BufferedInputStream.java:265) java.io.DataInputStream.readShort(DataInputStream.java:312) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:67) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:269) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: hconnection-0x48646271-shared-pool-11 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-4-7 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-4-4 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: hconnection-0x48646271-shared-pool-13 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-3 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: hconnection-0x48646271-shared-pool-10 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: Timer for 'HBase' metrics system java.lang.Object.wait(Native Method) java.util.TimerThread.mainLoop(Timer.java:552) java.util.TimerThread.run(Timer.java:505) Potentially hanging thread: RS:0;jenkins-hbase20:34681-shortCompactions-0 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.PriorityBlockingQueue.take(PriorityBlockingQueue.java:549) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-4-9 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:458) java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362) java.util.concurrent.SynchronousQueue.take(SynchronousQueue.java:924) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_-2053686367_15 at /127.0.0.1:44374 [Waiting for operation #3] sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:246) java.io.BufferedInputStream.read(BufferedInputStream.java:265) java.io.DataInputStream.readShort(DataInputStream.java:312) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:67) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:269) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-4-3 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: HFileArchiver-3 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-9 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-8 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-2 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-4-13 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) - Thread LEAK? -, OpenFileDescriptor=560 (was 470) - OpenFileDescriptor LEAK? -, MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=397 (was 413), ProcessCount=169 (was 169), AvailableMemoryMB=5901 (was 6340) 2023-04-15 02:54:00,180 INFO [Listener at localhost.localdomain/33693] hbase.ResourceChecker(147): before: TestAcidGuaranteesWithEagerPolicy#testMobMixedAtomicity Thread=234, OpenFileDescriptor=560, MaxFileDescriptor=60000, SystemLoadAverage=397, ProcessCount=169, AvailableMemoryMB=5901 2023-04-15 02:54:00,182 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] util.TableDescriptorChecker(340): MEMSTORE_FLUSHSIZE for table descriptor or "hbase.hregion.memstore.flush.size" (131072) is too small, which might cause very frequent flushing. 2023-04-15 02:54:00,182 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.HMaster$4(2112): Client=jenkins//148.251.75.209 create 'TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-04-15 02:54:00,183 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] procedure2.ProcedureExecutor(1029): Stored pid=16, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION; CreateTableProcedure table=TestAcidGuarantees 2023-04-15 02:54:00,185 INFO [PEWorker-5] procedure.CreateTableProcedure(80): pid=16, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_PRE_OPERATION 2023-04-15 02:54:00,185 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(697): Client=jenkins//148.251.75.209 procedure request for creating table: namespace: "default" qualifier: "TestAcidGuarantees" procId is: 16 2023-04-15 02:54:00,186 INFO [PEWorker-5] procedure.CreateTableProcedure(80): pid=16, state=RUNNABLE:CREATE_TABLE_WRITE_FS_LAYOUT, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_WRITE_FS_LAYOUT 2023-04-15 02:54:00,186 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(1227): Checking to see if procedure is done pid=16 2023-04-15 02:54:00,187 DEBUG [HFileArchiver-4] backup.HFileArchiver(131): ARCHIVING hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/.tmp/data/default/TestAcidGuarantees/b7b06ed62cbe8eca9103c3d111564e6c 2023-04-15 02:54:00,188 DEBUG [HFileArchiver-4] backup.HFileArchiver(153): Directory hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/.tmp/data/default/TestAcidGuarantees/b7b06ed62cbe8eca9103c3d111564e6c empty. 2023-04-15 02:54:00,188 DEBUG [HFileArchiver-4] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/.tmp/data/default/TestAcidGuarantees/b7b06ed62cbe8eca9103c3d111564e6c 2023-04-15 02:54:00,188 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(328): Archived TestAcidGuarantees regions 2023-04-15 02:54:00,288 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(1227): Checking to see if procedure is done pid=16 2023-04-15 02:54:00,489 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(1227): Checking to see if procedure is done pid=16 2023-04-15 02:54:00,602 DEBUG [PEWorker-5] util.FSTableDescriptors(570): Wrote into hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/.tmp/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000001 2023-04-15 02:54:00,605 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(7675): creating {ENCODED => b7b06ed62cbe8eca9103c3d111564e6c, NAME => 'TestAcidGuarantees,,1681527240182.b7b06ed62cbe8eca9103c3d111564e6c.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/.tmp 2023-04-15 02:54:00,791 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(1227): Checking to see if procedure is done pid=16 2023-04-15 02:54:01,017 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1681527240182.b7b06ed62cbe8eca9103c3d111564e6c.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-15 02:54:01,017 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1604): Closing b7b06ed62cbe8eca9103c3d111564e6c, disabling compactions & flushes 2023-04-15 02:54:01,018 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1681527240182.b7b06ed62cbe8eca9103c3d111564e6c. 2023-04-15 02:54:01,018 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1681527240182.b7b06ed62cbe8eca9103c3d111564e6c. 2023-04-15 02:54:01,018 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1681527240182.b7b06ed62cbe8eca9103c3d111564e6c. after waiting 0 ms 2023-04-15 02:54:01,018 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1681527240182.b7b06ed62cbe8eca9103c3d111564e6c. 2023-04-15 02:54:01,018 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1681527240182.b7b06ed62cbe8eca9103c3d111564e6c. 2023-04-15 02:54:01,018 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1558): Region close journal for b7b06ed62cbe8eca9103c3d111564e6c: 2023-04-15 02:54:01,024 INFO [PEWorker-5] procedure.CreateTableProcedure(80): pid=16, state=RUNNABLE:CREATE_TABLE_ADD_TO_META, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_ADD_TO_META 2023-04-15 02:54:01,026 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1681527240182.b7b06ed62cbe8eca9103c3d111564e6c.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1681527241026"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1681527241026"}]},"ts":"1681527241026"} 2023-04-15 02:54:01,028 INFO [PEWorker-5] hbase.MetaTableAccessor(1496): Added 1 regions to meta. 2023-04-15 02:54:01,029 INFO [PEWorker-5] procedure.CreateTableProcedure(80): pid=16, state=RUNNABLE:CREATE_TABLE_ASSIGN_REGIONS, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_ASSIGN_REGIONS 2023-04-15 02:54:01,030 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1681527241030"}]},"ts":"1681527241030"} 2023-04-15 02:54:01,031 INFO [PEWorker-5] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=ENABLING in hbase:meta 2023-04-15 02:54:01,035 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=17, ppid=16, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=TestAcidGuarantees, region=b7b06ed62cbe8eca9103c3d111564e6c, ASSIGN}] 2023-04-15 02:54:01,037 INFO [PEWorker-2] procedure.MasterProcedureScheduler(727): Took xlock for pid=17, ppid=16, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=TestAcidGuarantees, region=b7b06ed62cbe8eca9103c3d111564e6c, ASSIGN 2023-04-15 02:54:01,038 INFO [PEWorker-2] assignment.TransitRegionStateProcedure(193): Starting pid=17, ppid=16, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=TestAcidGuarantees, region=b7b06ed62cbe8eca9103c3d111564e6c, ASSIGN; state=OFFLINE, location=jenkins-hbase20.apache.org,34681,1681527209133; forceNewPlan=false, retain=false 2023-04-15 02:54:01,190 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=17 updating hbase:meta row=b7b06ed62cbe8eca9103c3d111564e6c, regionState=OPENING, regionLocation=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:01,190 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1681527240182.b7b06ed62cbe8eca9103c3d111564e6c.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1681527241190"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1681527241190"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1681527241190"}]},"ts":"1681527241190"} 2023-04-15 02:54:01,194 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=18, ppid=17, state=RUNNABLE; OpenRegionProcedure b7b06ed62cbe8eca9103c3d111564e6c, server=jenkins-hbase20.apache.org,34681,1681527209133}] 2023-04-15 02:54:01,293 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(1227): Checking to see if procedure is done pid=16 2023-04-15 02:54:01,355 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] handler.AssignRegionHandler(128): Open TestAcidGuarantees,,1681527240182.b7b06ed62cbe8eca9103c3d111564e6c. 2023-04-15 02:54:01,355 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => b7b06ed62cbe8eca9103c3d111564e6c, NAME => 'TestAcidGuarantees,,1681527240182.b7b06ed62cbe8eca9103c3d111564e6c.', STARTKEY => '', ENDKEY => ''} 2023-04-15 02:54:01,356 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table TestAcidGuarantees b7b06ed62cbe8eca9103c3d111564e6c 2023-04-15 02:54:01,356 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1681527240182.b7b06ed62cbe8eca9103c3d111564e6c.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-15 02:54:01,356 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7894): checking encryption for b7b06ed62cbe8eca9103c3d111564e6c 2023-04-15 02:54:01,356 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7897): checking classloading for b7b06ed62cbe8eca9103c3d111564e6c 2023-04-15 02:54:01,358 INFO [StoreOpener-b7b06ed62cbe8eca9103c3d111564e6c-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family A of region b7b06ed62cbe8eca9103c3d111564e6c 2023-04-15 02:54:01,359 DEBUG [StoreOpener-b7b06ed62cbe8eca9103c3d111564e6c-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/b7b06ed62cbe8eca9103c3d111564e6c/A 2023-04-15 02:54:01,360 DEBUG [StoreOpener-b7b06ed62cbe8eca9103c3d111564e6c-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/b7b06ed62cbe8eca9103c3d111564e6c/A 2023-04-15 02:54:01,360 INFO [StoreOpener-b7b06ed62cbe8eca9103c3d111564e6c-1] regionserver.CompactingMemStore(121): Store=A, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-15 02:54:01,360 INFO [StoreOpener-b7b06ed62cbe8eca9103c3d111564e6c-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region b7b06ed62cbe8eca9103c3d111564e6c columnFamilyName A 2023-04-15 02:54:01,361 INFO [StoreOpener-b7b06ed62cbe8eca9103c3d111564e6c-1] regionserver.HStore(310): Store=b7b06ed62cbe8eca9103c3d111564e6c/A, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-15 02:54:01,361 INFO [StoreOpener-b7b06ed62cbe8eca9103c3d111564e6c-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family B of region b7b06ed62cbe8eca9103c3d111564e6c 2023-04-15 02:54:01,362 DEBUG [StoreOpener-b7b06ed62cbe8eca9103c3d111564e6c-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/b7b06ed62cbe8eca9103c3d111564e6c/B 2023-04-15 02:54:01,362 DEBUG [StoreOpener-b7b06ed62cbe8eca9103c3d111564e6c-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/b7b06ed62cbe8eca9103c3d111564e6c/B 2023-04-15 02:54:01,362 INFO [StoreOpener-b7b06ed62cbe8eca9103c3d111564e6c-1] regionserver.CompactingMemStore(121): Store=B, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-15 02:54:01,362 INFO [StoreOpener-b7b06ed62cbe8eca9103c3d111564e6c-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region b7b06ed62cbe8eca9103c3d111564e6c columnFamilyName B 2023-04-15 02:54:01,363 INFO [StoreOpener-b7b06ed62cbe8eca9103c3d111564e6c-1] regionserver.HStore(310): Store=b7b06ed62cbe8eca9103c3d111564e6c/B, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-15 02:54:01,363 INFO [StoreOpener-b7b06ed62cbe8eca9103c3d111564e6c-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family C of region b7b06ed62cbe8eca9103c3d111564e6c 2023-04-15 02:54:01,364 DEBUG [StoreOpener-b7b06ed62cbe8eca9103c3d111564e6c-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/b7b06ed62cbe8eca9103c3d111564e6c/C 2023-04-15 02:54:01,364 DEBUG [StoreOpener-b7b06ed62cbe8eca9103c3d111564e6c-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/b7b06ed62cbe8eca9103c3d111564e6c/C 2023-04-15 02:54:01,364 INFO [StoreOpener-b7b06ed62cbe8eca9103c3d111564e6c-1] regionserver.CompactingMemStore(121): Store=C, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-15 02:54:01,364 INFO [StoreOpener-b7b06ed62cbe8eca9103c3d111564e6c-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region b7b06ed62cbe8eca9103c3d111564e6c columnFamilyName C 2023-04-15 02:54:01,364 INFO [StoreOpener-b7b06ed62cbe8eca9103c3d111564e6c-1] regionserver.HStore(310): Store=b7b06ed62cbe8eca9103c3d111564e6c/C, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-15 02:54:01,365 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1146): Setting FlushNonSloppyStoresFirstPolicy for the region=TestAcidGuarantees,,1681527240182.b7b06ed62cbe8eca9103c3d111564e6c. 2023-04-15 02:54:01,365 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/b7b06ed62cbe8eca9103c3d111564e6c 2023-04-15 02:54:01,366 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/b7b06ed62cbe8eca9103c3d111564e6c 2023-04-15 02:54:01,367 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table TestAcidGuarantees descriptor;using region.getMemStoreFlushHeapSize/# of families (16.0 M)) instead. 2023-04-15 02:54:01,368 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1055): writing seq id for b7b06ed62cbe8eca9103c3d111564e6c 2023-04-15 02:54:01,370 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/b7b06ed62cbe8eca9103c3d111564e6c/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-04-15 02:54:01,370 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1072): Opened b7b06ed62cbe8eca9103c3d111564e6c; next sequenceid=2; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=62681261, jitterRate=-0.06597642600536346}, FlushLargeStoresPolicy{flushSizeLowerBound=16777216} 2023-04-15 02:54:01,370 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(965): Region open journal for b7b06ed62cbe8eca9103c3d111564e6c: 2023-04-15 02:54:01,371 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for TestAcidGuarantees,,1681527240182.b7b06ed62cbe8eca9103c3d111564e6c., pid=18, masterSystemTime=1681527241348 2023-04-15 02:54:01,372 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for TestAcidGuarantees,,1681527240182.b7b06ed62cbe8eca9103c3d111564e6c. 2023-04-15 02:54:01,372 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] handler.AssignRegionHandler(156): Opened TestAcidGuarantees,,1681527240182.b7b06ed62cbe8eca9103c3d111564e6c. 2023-04-15 02:54:01,373 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=17 updating hbase:meta row=b7b06ed62cbe8eca9103c3d111564e6c, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:01,373 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"TestAcidGuarantees,,1681527240182.b7b06ed62cbe8eca9103c3d111564e6c.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1681527241373"},{"qualifier":"server","vlen":32,"tag":[],"timestamp":"1681527241373"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1681527241373"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1681527241373"}]},"ts":"1681527241373"} 2023-04-15 02:54:01,376 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=18, resume processing ppid=17 2023-04-15 02:54:01,376 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=18, ppid=17, state=SUCCESS; OpenRegionProcedure b7b06ed62cbe8eca9103c3d111564e6c, server=jenkins-hbase20.apache.org,34681,1681527209133 in 180 msec 2023-04-15 02:54:01,377 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=17, resume processing ppid=16 2023-04-15 02:54:01,377 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=17, ppid=16, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=b7b06ed62cbe8eca9103c3d111564e6c, ASSIGN in 341 msec 2023-04-15 02:54:01,378 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=16, state=RUNNABLE:CREATE_TABLE_UPDATE_DESC_CACHE, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_UPDATE_DESC_CACHE 2023-04-15 02:54:01,378 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1681527241378"}]},"ts":"1681527241378"} 2023-04-15 02:54:01,379 INFO [PEWorker-2] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=ENABLED in hbase:meta 2023-04-15 02:54:01,401 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=16, state=RUNNABLE:CREATE_TABLE_POST_OPERATION, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_POST_OPERATION 2023-04-15 02:54:01,402 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=16, state=SUCCESS; CreateTableProcedure table=TestAcidGuarantees in 1.2190 sec 2023-04-15 02:54:02,296 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(1227): Checking to see if procedure is done pid=16 2023-04-15 02:54:02,297 INFO [Listener at localhost.localdomain/33693] client.HBaseAdmin$TableFuture(3541): Operation: CREATE, Table Name: default:TestAcidGuarantees, procId: 16 completed 2023-04-15 02:54:02,299 DEBUG [Listener at localhost.localdomain/33693] zookeeper.ReadOnlyZKClient(139): Connect 0x35bef85d to 127.0.0.1:51762 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-15 02:54:02,306 DEBUG [Listener at localhost.localdomain/33693] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@3105e416, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-15 02:54:02,309 DEBUG [Listener at localhost.localdomain/33693] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-15 02:54:02,317 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:39070, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-15 02:54:02,319 DEBUG [Listener at localhost.localdomain/33693] ipc.RpcConnection(124): Using SIMPLE authentication for service=MasterService, sasl=false 2023-04-15 02:54:02,323 INFO [RS-EventLoopGroup-1-2] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:45800, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=MasterService 2023-04-15 02:54:02,330 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] util.TableDescriptorChecker(340): MEMSTORE_FLUSHSIZE for table descriptor or "hbase.hregion.memstore.flush.size" (131072) is too small, which might cause very frequent flushing. 2023-04-15 02:54:02,330 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.HMaster$12(2454): Client=jenkins//148.251.75.209 modify table TestAcidGuarantees from 'TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} to 'TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', MOB_THRESHOLD => '4', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-04-15 02:54:02,340 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] procedure2.ProcedureExecutor(1029): Stored pid=19, state=RUNNABLE:MODIFY_TABLE_PREPARE; ModifyTableProcedure table=TestAcidGuarantees 2023-04-15 02:54:02,754 DEBUG [PEWorker-1] util.FSTableDescriptors(570): Wrote into hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000002 2023-04-15 02:54:02,756 DEBUG [PEWorker-1] util.FSTableDescriptors(519): Deleted hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000001 2023-04-15 02:54:02,756 INFO [PEWorker-1] util.FSTableDescriptors(293): Updated tableinfo=hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000002 2023-04-15 02:54:02,763 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=20, ppid=19, state=RUNNABLE:REOPEN_TABLE_REGIONS_GET_REGIONS; ReopenTableRegionsProcedure table=TestAcidGuarantees}] 2023-04-15 02:54:02,775 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=21, ppid=20, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=b7b06ed62cbe8eca9103c3d111564e6c, REOPEN/MOVE}] 2023-04-15 02:54:02,776 INFO [PEWorker-3] procedure.MasterProcedureScheduler(727): Took xlock for pid=21, ppid=20, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=b7b06ed62cbe8eca9103c3d111564e6c, REOPEN/MOVE 2023-04-15 02:54:02,776 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=21 updating hbase:meta row=b7b06ed62cbe8eca9103c3d111564e6c, regionState=CLOSING, regionLocation=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:02,777 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1681527240182.b7b06ed62cbe8eca9103c3d111564e6c.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1681527242776"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1681527242776"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1681527242776"}]},"ts":"1681527242776"} 2023-04-15 02:54:02,778 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=22, ppid=21, state=RUNNABLE; CloseRegionProcedure b7b06ed62cbe8eca9103c3d111564e6c, server=jenkins-hbase20.apache.org,34681,1681527209133}] 2023-04-15 02:54:02,930 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] handler.UnassignRegionHandler(109): Close b7b06ed62cbe8eca9103c3d111564e6c 2023-04-15 02:54:02,930 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1604): Closing b7b06ed62cbe8eca9103c3d111564e6c, disabling compactions & flushes 2023-04-15 02:54:02,931 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1681527240182.b7b06ed62cbe8eca9103c3d111564e6c. 2023-04-15 02:54:02,931 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1681527240182.b7b06ed62cbe8eca9103c3d111564e6c. 2023-04-15 02:54:02,931 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1681527240182.b7b06ed62cbe8eca9103c3d111564e6c. after waiting 0 ms 2023-04-15 02:54:02,931 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1681527240182.b7b06ed62cbe8eca9103c3d111564e6c. 2023-04-15 02:54:02,935 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/b7b06ed62cbe8eca9103c3d111564e6c/recovered.edits/4.seqid, newMaxSeqId=4, maxSeqId=1 2023-04-15 02:54:02,936 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1681527240182.b7b06ed62cbe8eca9103c3d111564e6c. 2023-04-15 02:54:02,936 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1558): Region close journal for b7b06ed62cbe8eca9103c3d111564e6c: 2023-04-15 02:54:02,936 WARN [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionServer(3509): Not adding moved region record: b7b06ed62cbe8eca9103c3d111564e6c to self. 2023-04-15 02:54:02,938 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] handler.UnassignRegionHandler(147): Closed b7b06ed62cbe8eca9103c3d111564e6c 2023-04-15 02:54:02,939 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=21 updating hbase:meta row=b7b06ed62cbe8eca9103c3d111564e6c, regionState=CLOSED 2023-04-15 02:54:02,939 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1681527240182.b7b06ed62cbe8eca9103c3d111564e6c.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1681527242939"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1681527242939"}]},"ts":"1681527242939"} 2023-04-15 02:54:02,944 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=22, resume processing ppid=21 2023-04-15 02:54:02,944 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=22, ppid=21, state=SUCCESS; CloseRegionProcedure b7b06ed62cbe8eca9103c3d111564e6c, server=jenkins-hbase20.apache.org,34681,1681527209133 in 163 msec 2023-04-15 02:54:02,944 INFO [PEWorker-1] assignment.TransitRegionStateProcedure(193): Starting pid=21, ppid=20, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=TestAcidGuarantees, region=b7b06ed62cbe8eca9103c3d111564e6c, REOPEN/MOVE; state=CLOSED, location=jenkins-hbase20.apache.org,34681,1681527209133; forceNewPlan=false, retain=true 2023-04-15 02:54:03,095 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=21 updating hbase:meta row=b7b06ed62cbe8eca9103c3d111564e6c, regionState=OPENING, regionLocation=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:03,096 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1681527240182.b7b06ed62cbe8eca9103c3d111564e6c.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1681527243095"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1681527243095"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1681527243095"}]},"ts":"1681527243095"} 2023-04-15 02:54:03,099 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=23, ppid=21, state=RUNNABLE; OpenRegionProcedure b7b06ed62cbe8eca9103c3d111564e6c, server=jenkins-hbase20.apache.org,34681,1681527209133}] 2023-04-15 02:54:03,260 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] handler.AssignRegionHandler(128): Open TestAcidGuarantees,,1681527240182.b7b06ed62cbe8eca9103c3d111564e6c. 2023-04-15 02:54:03,260 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => b7b06ed62cbe8eca9103c3d111564e6c, NAME => 'TestAcidGuarantees,,1681527240182.b7b06ed62cbe8eca9103c3d111564e6c.', STARTKEY => '', ENDKEY => ''} 2023-04-15 02:54:03,261 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table TestAcidGuarantees b7b06ed62cbe8eca9103c3d111564e6c 2023-04-15 02:54:03,261 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1681527240182.b7b06ed62cbe8eca9103c3d111564e6c.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-15 02:54:03,261 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7894): checking encryption for b7b06ed62cbe8eca9103c3d111564e6c 2023-04-15 02:54:03,262 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7897): checking classloading for b7b06ed62cbe8eca9103c3d111564e6c 2023-04-15 02:54:03,265 INFO [StoreOpener-b7b06ed62cbe8eca9103c3d111564e6c-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family A of region b7b06ed62cbe8eca9103c3d111564e6c 2023-04-15 02:54:03,267 DEBUG [StoreOpener-b7b06ed62cbe8eca9103c3d111564e6c-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/b7b06ed62cbe8eca9103c3d111564e6c/A 2023-04-15 02:54:03,267 DEBUG [StoreOpener-b7b06ed62cbe8eca9103c3d111564e6c-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/b7b06ed62cbe8eca9103c3d111564e6c/A 2023-04-15 02:54:03,267 INFO [StoreOpener-b7b06ed62cbe8eca9103c3d111564e6c-1] regionserver.CompactingMemStore(121): Store=A, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-15 02:54:03,270 INFO [StoreOpener-b7b06ed62cbe8eca9103c3d111564e6c-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region b7b06ed62cbe8eca9103c3d111564e6c columnFamilyName A 2023-04-15 02:54:03,273 INFO [StoreOpener-b7b06ed62cbe8eca9103c3d111564e6c-1] regionserver.HStore(310): Store=b7b06ed62cbe8eca9103c3d111564e6c/A, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-15 02:54:03,274 INFO [StoreOpener-b7b06ed62cbe8eca9103c3d111564e6c-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family B of region b7b06ed62cbe8eca9103c3d111564e6c 2023-04-15 02:54:03,276 DEBUG [StoreOpener-b7b06ed62cbe8eca9103c3d111564e6c-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/b7b06ed62cbe8eca9103c3d111564e6c/B 2023-04-15 02:54:03,276 DEBUG [StoreOpener-b7b06ed62cbe8eca9103c3d111564e6c-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/b7b06ed62cbe8eca9103c3d111564e6c/B 2023-04-15 02:54:03,277 INFO [StoreOpener-b7b06ed62cbe8eca9103c3d111564e6c-1] regionserver.CompactingMemStore(121): Store=B, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-15 02:54:03,278 INFO [StoreOpener-b7b06ed62cbe8eca9103c3d111564e6c-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region b7b06ed62cbe8eca9103c3d111564e6c columnFamilyName B 2023-04-15 02:54:03,278 INFO [StoreOpener-b7b06ed62cbe8eca9103c3d111564e6c-1] regionserver.HStore(310): Store=b7b06ed62cbe8eca9103c3d111564e6c/B, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-15 02:54:03,278 INFO [StoreOpener-b7b06ed62cbe8eca9103c3d111564e6c-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family C of region b7b06ed62cbe8eca9103c3d111564e6c 2023-04-15 02:54:03,279 DEBUG [StoreOpener-b7b06ed62cbe8eca9103c3d111564e6c-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/b7b06ed62cbe8eca9103c3d111564e6c/C 2023-04-15 02:54:03,279 DEBUG [StoreOpener-b7b06ed62cbe8eca9103c3d111564e6c-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/b7b06ed62cbe8eca9103c3d111564e6c/C 2023-04-15 02:54:03,280 INFO [StoreOpener-b7b06ed62cbe8eca9103c3d111564e6c-1] regionserver.CompactingMemStore(121): Store=C, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-15 02:54:03,280 INFO [StoreOpener-b7b06ed62cbe8eca9103c3d111564e6c-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region b7b06ed62cbe8eca9103c3d111564e6c columnFamilyName C 2023-04-15 02:54:03,281 INFO [StoreOpener-b7b06ed62cbe8eca9103c3d111564e6c-1] regionserver.HStore(310): Store=b7b06ed62cbe8eca9103c3d111564e6c/C, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-15 02:54:03,281 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1146): Setting FlushNonSloppyStoresFirstPolicy for the region=TestAcidGuarantees,,1681527240182.b7b06ed62cbe8eca9103c3d111564e6c. 2023-04-15 02:54:03,282 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/b7b06ed62cbe8eca9103c3d111564e6c 2023-04-15 02:54:03,283 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/b7b06ed62cbe8eca9103c3d111564e6c 2023-04-15 02:54:03,285 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table TestAcidGuarantees descriptor;using region.getMemStoreFlushHeapSize/# of families (16.0 M)) instead. 2023-04-15 02:54:03,286 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1055): writing seq id for b7b06ed62cbe8eca9103c3d111564e6c 2023-04-15 02:54:03,287 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1072): Opened b7b06ed62cbe8eca9103c3d111564e6c; next sequenceid=5; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=64045598, jitterRate=-0.045646220445632935}, FlushLargeStoresPolicy{flushSizeLowerBound=16777216} 2023-04-15 02:54:03,287 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(965): Region open journal for b7b06ed62cbe8eca9103c3d111564e6c: 2023-04-15 02:54:03,287 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for TestAcidGuarantees,,1681527240182.b7b06ed62cbe8eca9103c3d111564e6c., pid=23, masterSystemTime=1681527243253 2023-04-15 02:54:03,289 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for TestAcidGuarantees,,1681527240182.b7b06ed62cbe8eca9103c3d111564e6c. 2023-04-15 02:54:03,289 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] handler.AssignRegionHandler(156): Opened TestAcidGuarantees,,1681527240182.b7b06ed62cbe8eca9103c3d111564e6c. 2023-04-15 02:54:03,289 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=21 updating hbase:meta row=b7b06ed62cbe8eca9103c3d111564e6c, regionState=OPEN, openSeqNum=5, regionLocation=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:03,289 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"TestAcidGuarantees,,1681527240182.b7b06ed62cbe8eca9103c3d111564e6c.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1681527243289"},{"qualifier":"server","vlen":32,"tag":[],"timestamp":"1681527243289"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1681527243289"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1681527243289"}]},"ts":"1681527243289"} 2023-04-15 02:54:03,293 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=23, resume processing ppid=21 2023-04-15 02:54:03,293 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=23, ppid=21, state=SUCCESS; OpenRegionProcedure b7b06ed62cbe8eca9103c3d111564e6c, server=jenkins-hbase20.apache.org,34681,1681527209133 in 192 msec 2023-04-15 02:54:03,295 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=21, resume processing ppid=20 2023-04-15 02:54:03,296 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=21, ppid=20, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=b7b06ed62cbe8eca9103c3d111564e6c, REOPEN/MOVE in 518 msec 2023-04-15 02:54:03,299 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=20, resume processing ppid=19 2023-04-15 02:54:03,300 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=20, ppid=19, state=SUCCESS; ReopenTableRegionsProcedure table=TestAcidGuarantees in 536 msec 2023-04-15 02:54:03,303 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=19, state=SUCCESS; ModifyTableProcedure table=TestAcidGuarantees in 967 msec 2023-04-15 02:54:03,307 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(1227): Checking to see if procedure is done pid=19 2023-04-15 02:54:03,315 DEBUG [Listener at localhost.localdomain/33693] zookeeper.ReadOnlyZKClient(139): Connect 0x071f9a8a to 127.0.0.1:51762 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-15 02:54:03,320 DEBUG [Listener at localhost.localdomain/33693] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@4ef5b30d, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-15 02:54:03,320 DEBUG [Listener at localhost.localdomain/33693] zookeeper.ReadOnlyZKClient(139): Connect 0x430c7474 to 127.0.0.1:51762 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-15 02:54:03,325 DEBUG [Listener at localhost.localdomain/33693] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@3b75e45b, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-15 02:54:03,325 DEBUG [Listener at localhost.localdomain/33693] zookeeper.ReadOnlyZKClient(139): Connect 0x2d781def to 127.0.0.1:51762 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-15 02:54:03,336 DEBUG [Listener at localhost.localdomain/33693] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@df763e0, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-15 02:54:03,336 DEBUG [Listener at localhost.localdomain/33693] zookeeper.ReadOnlyZKClient(139): Connect 0x5649f410 to 127.0.0.1:51762 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-15 02:54:03,344 DEBUG [Listener at localhost.localdomain/33693] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@6017755a, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-15 02:54:03,345 DEBUG [Listener at localhost.localdomain/33693] zookeeper.ReadOnlyZKClient(139): Connect 0x7acc1387 to 127.0.0.1:51762 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-15 02:54:03,352 DEBUG [Listener at localhost.localdomain/33693] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@633c81ba, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-15 02:54:03,353 DEBUG [Listener at localhost.localdomain/33693] zookeeper.ReadOnlyZKClient(139): Connect 0x61ad5d1c to 127.0.0.1:51762 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-15 02:54:03,361 DEBUG [Listener at localhost.localdomain/33693] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@661b21c2, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-15 02:54:03,361 DEBUG [Listener at localhost.localdomain/33693] zookeeper.ReadOnlyZKClient(139): Connect 0x486fba63 to 127.0.0.1:51762 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-15 02:54:03,365 DEBUG [Listener at localhost.localdomain/33693] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@43a14bcf, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-15 02:54:03,366 DEBUG [Listener at localhost.localdomain/33693] zookeeper.ReadOnlyZKClient(139): Connect 0x50feca84 to 127.0.0.1:51762 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-15 02:54:03,378 DEBUG [Listener at localhost.localdomain/33693] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@6577ce5a, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-15 02:54:03,378 DEBUG [Listener at localhost.localdomain/33693] zookeeper.ReadOnlyZKClient(139): Connect 0x0bd79dff to 127.0.0.1:51762 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-15 02:54:03,382 DEBUG [Listener at localhost.localdomain/33693] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@2d5bdceb, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-15 02:54:03,392 DEBUG [hconnection-0x6261ebe4-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-15 02:54:03,392 DEBUG [hconnection-0x565cb857-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-15 02:54:03,392 DEBUG [hconnection-0x7d3b185f-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-15 02:54:03,394 DEBUG [hconnection-0x6bacfdd6-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-15 02:54:03,395 DEBUG [hconnection-0x22aac745-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-15 02:54:03,395 DEBUG [hconnection-0x12a766fc-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-15 02:54:03,395 DEBUG [hconnection-0x546c01fd-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-15 02:54:03,397 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:39072, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-15 02:54:03,397 DEBUG [hconnection-0x9ae667d-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-15 02:54:03,398 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:39106, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-15 02:54:03,398 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:39116, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-15 02:54:03,405 DEBUG [hconnection-0x63efb28a-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-15 02:54:03,405 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:39086, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-15 02:54:03,405 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:39102, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-15 02:54:03,405 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:39108, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-15 02:54:03,406 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:39132, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-15 02:54:03,405 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:39114, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-15 02:54:03,408 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-15 02:54:03,409 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:39142, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-15 02:54:03,419 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-15 02:54:03,419 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-15 02:54:03,419 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-15 02:54:03,419 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-15 02:54:03,419 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-15 02:54:03,419 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-15 02:54:03,419 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:03,419 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:54:03,425 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-15 02:54:03,425 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-15 02:54:03,425 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:03,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-15 02:54:03,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:54:03,426 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:03,426 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-15 02:54:03,426 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:03,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-15 02:54:03,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:54:03,426 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-15 02:54:03,426 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-15 02:54:03,426 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-15 02:54:03,426 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-15 02:54:03,427 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-15 02:54:03,427 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-15 02:54:03,433 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-15 02:54:03,433 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-15 02:54:03,433 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681527240182.b7b06ed62cbe8eca9103c3d111564e6c. 2023-04-15 02:54:03,433 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681527240182.b7b06ed62cbe8eca9103c3d111564e6c. started... 2023-04-15 02:54:03,434 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34681] regionserver.HRegion(9158): Flush requested on b7b06ed62cbe8eca9103c3d111564e6c 2023-04-15 02:54:03,435 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681527240182.b7b06ed62cbe8eca9103c3d111564e6c. as already flushing 2023-04-15 02:54:03,435 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing b7b06ed62cbe8eca9103c3d111564e6c 3/3 column families, dataSize=53.67 KB heapSize=160.13 KB 2023-04-15 02:54:03,435 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681527240182.b7b06ed62cbe8eca9103c3d111564e6c. 2023-04-15 02:54:03,435 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-15 02:54:03,435 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-15 02:54:03,435 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:03,436 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK b7b06ed62cbe8eca9103c3d111564e6c, store=A 2023-04-15 02:54:03,436 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-15 02:54:03,436 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK b7b06ed62cbe8eca9103c3d111564e6c, store=B 2023-04-15 02:54:03,436 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-15 02:54:03,436 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK b7b06ed62cbe8eca9103c3d111564e6c, store=C 2023-04-15 02:54:03,436 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-15 02:54:03,446 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-15 02:54:03,446 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34681,1681527209133' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-15 02:54:03,449 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:03,449 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:03,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:03,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:54:03,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:54:03,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:54:03,449 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:03,449 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-15 02:54:03,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:54:03,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:54:03,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:03,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:54:03,452 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34681,1681527209133' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-15 02:54:03,452 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@728b186e[Count = 0] remaining members to acquire global barrier 2023-04-15 02:54:03,452 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-15 02:54:03,452 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:03,453 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:03,453 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:03,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:03,453 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-15 02:54:03,453 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:03,453 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-15 02:54:03,453 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-15 02:54:03,453 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34681,1681527209133' in zk 2023-04-15 02:54:03,454 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:03,454 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-15 02:54:03,454 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-15 02:54:03,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:03,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:54:03,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:54:03,454 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-15 02:54:03,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:54:03,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:54:03,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:54:03,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:03,456 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b7b06ed62cbe8eca9103c3d111564e6c, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:54:03,460 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=309, cellSize=14145, totalHeapSize=32376, min timestamp=1681527243449, max timestamp=1681527243455Num uniques 859; 2023-04-15 02:54:03,460 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-2] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=14145, suffix heap size=117064,new segment heap size=32376 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=309 2023-04-15 02:54:03,460 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34681] ipc.CallRunner(144): callId: 7 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:39106 deadline: 1681527303453, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b7b06ed62cbe8eca9103c3d111564e6c, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:03,462 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=309, cellSize=14145, totalHeapSize=32376, min timestamp=1681527243449, max timestamp=1681527243455Num uniques 859; 2023-04-15 02:54:03,462 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-5] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=14145, suffix heap size=117064,new segment heap size=32376 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=309 2023-04-15 02:54:03,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:54:03,467 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=309, cellSize=14145, totalHeapSize=32376, min timestamp=1681527243449, max timestamp=1681527243455Num uniques 859; 2023-04-15 02:54:03,467 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-1] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=14145, suffix heap size=117064,new segment heap size=32376 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=309 2023-04-15 02:54:03,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:54:03,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:03,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34681,1681527209133': 2023-04-15 02:54:03,469 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34681,1681527209133' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-15 02:54:03,469 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-15 02:54:03,469 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-15 02:54:03,469 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-15 02:54:03,469 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-15 02:54:03,469 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-15 02:54:03,477 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:03,477 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:03,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:03,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:54:03,477 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:03,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:03,477 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-15 02:54:03,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:54:03,478 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-15 02:54:03,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-15 02:54:03,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:54:03,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:03,478 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:03,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:54:03,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:54:03,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:54:03,479 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:03,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:03,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:54:03,480 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b7b06ed62cbe8eca9103c3d111564e6c, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:54:03,480 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34681] ipc.CallRunner(144): callId: 11 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:39116 deadline: 1681527303478, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b7b06ed62cbe8eca9103c3d111564e6c, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:03,480 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b7b06ed62cbe8eca9103c3d111564e6c, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:54:03,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:54:03,480 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34681] ipc.CallRunner(144): callId: 12 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:39108 deadline: 1681527303478, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b7b06ed62cbe8eca9103c3d111564e6c, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:03,481 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b7b06ed62cbe8eca9103c3d111564e6c, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:54:03,481 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34681] ipc.CallRunner(144): callId: 11 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:39132 deadline: 1681527303480, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b7b06ed62cbe8eca9103c3d111564e6c, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:03,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:03,483 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b7b06ed62cbe8eca9103c3d111564e6c, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:54:03,483 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34681] ipc.CallRunner(144): callId: 9 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:39142 deadline: 1681527303482, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b7b06ed62cbe8eca9103c3d111564e6c, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:03,485 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:03,485 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:54:03,485 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-15 02:54:03,485 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-15 02:54:03,485 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-15 02:54:03,485 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-15 02:54:03,485 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-15 02:54:03,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-15 02:54:03,485 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:54:03,485 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-15 02:54:03,486 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:03,486 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:03,486 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:03,486 DEBUG [Thread-864] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-15 02:54:03,486 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-15 02:54:03,486 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:03,486 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-15 02:54:03,486 DEBUG [Thread-864] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-15 02:54:03,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-15 02:54:03,501 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304157131d7079c734f1097b575fba3f782d0 to hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304157131d7079c734f1097b575fba3f782d0 2023-04-15 02:54:03,526 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=16, memsize=20.1 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/b7b06ed62cbe8eca9103c3d111564e6c/.tmp/A/fc91b7d3afe3454099e64a7900c7f638 2023-04-15 02:54:03,562 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=16 (bloomFilter=true), to=hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/b7b06ed62cbe8eca9103c3d111564e6c/.tmp/B/1898232003dd448791f6d6b222731e5a 2023-04-15 02:54:03,571 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b7b06ed62cbe8eca9103c3d111564e6c, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:54:03,572 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34681] ipc.CallRunner(144): callId: 9 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:39106 deadline: 1681527303566, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b7b06ed62cbe8eca9103c3d111564e6c, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:03,583 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b7b06ed62cbe8eca9103c3d111564e6c, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:54:03,584 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34681] ipc.CallRunner(144): callId: 14 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:39108 deadline: 1681527303583, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b7b06ed62cbe8eca9103c3d111564e6c, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:03,584 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b7b06ed62cbe8eca9103c3d111564e6c, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:54:03,584 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34681] ipc.CallRunner(144): callId: 13 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:39132 deadline: 1681527303584, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b7b06ed62cbe8eca9103c3d111564e6c, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:03,584 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b7b06ed62cbe8eca9103c3d111564e6c, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:54:03,585 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b7b06ed62cbe8eca9103c3d111564e6c, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:54:03,585 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34681] ipc.CallRunner(144): callId: 11 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:39142 deadline: 1681527303584, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b7b06ed62cbe8eca9103c3d111564e6c, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:03,585 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34681] ipc.CallRunner(144): callId: 13 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:39116 deadline: 1681527303584, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b7b06ed62cbe8eca9103c3d111564e6c, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:03,586 DEBUG [Thread-864] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-15 02:54:03,587 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-15 02:54:03,589 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-15 02:54:03,622 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-15 02:54:03,622 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-15 02:54:03,622 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-15 02:54:03,622 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-15 02:54:03,622 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-15 02:54:03,622 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-15 02:54:03,623 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:03,623 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:54:03,623 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-15 02:54:03,623 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:03,623 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-15 02:54:03,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-15 02:54:03,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:54:03,624 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:03,624 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-15 02:54:03,624 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:03,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-15 02:54:03,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:54:03,626 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-15 02:54:03,630 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-15 02:54:03,630 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-15 02:54:03,630 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-15 02:54:03,630 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-15 02:54:03,630 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-15 02:54:03,634 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-15 02:54:03,634 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-15 02:54:03,634 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681527240182.b7b06ed62cbe8eca9103c3d111564e6c. 2023-04-15 02:54:03,634 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681527240182.b7b06ed62cbe8eca9103c3d111564e6c. started... 2023-04-15 02:54:03,634 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681527240182.b7b06ed62cbe8eca9103c3d111564e6c. as already flushing 2023-04-15 02:54:03,634 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681527240182.b7b06ed62cbe8eca9103c3d111564e6c. 2023-04-15 02:54:03,634 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-15 02:54:03,634 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-15 02:54:03,634 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:03,634 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-15 02:54:03,634 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34681,1681527209133' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-15 02:54:03,636 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:03,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:03,636 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:03,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:54:03,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:54:03,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:54:03,637 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:03,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:54:03,637 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-15 02:54:03,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:54:03,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:03,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:54:03,638 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34681,1681527209133' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-15 02:54:03,638 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@31d8514[Count = 0] remaining members to acquire global barrier 2023-04-15 02:54:03,638 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-15 02:54:03,638 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:03,639 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:03,639 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:03,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:03,639 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-15 02:54:03,639 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-15 02:54:03,639 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34681,1681527209133' in zk 2023-04-15 02:54:03,639 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:03,639 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-15 02:54:03,640 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:03,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:03,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:54:03,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:54:03,640 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-15 02:54:03,640 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-15 02:54:03,640 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-15 02:54:03,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:54:03,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:54:03,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:54:03,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:03,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:54:03,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:54:03,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:03,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34681,1681527209133': 2023-04-15 02:54:03,643 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34681,1681527209133' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-15 02:54:03,643 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-15 02:54:03,643 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-15 02:54:03,643 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-15 02:54:03,643 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-15 02:54:03,643 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-15 02:54:03,652 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:03,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:03,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:54:03,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:54:03,652 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:03,652 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-15 02:54:03,652 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:03,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:03,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:54:03,652 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:03,653 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-15 02:54:03,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-15 02:54:03,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:54:03,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:03,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:54:03,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:54:03,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:03,654 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:03,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:54:03,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:54:03,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:03,656 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:03,656 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:54:03,656 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:54:03,656 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-15 02:54:03,656 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:03,657 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-15 02:54:03,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-15 02:54:03,656 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-15 02:54:03,657 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-15 02:54:03,657 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-15 02:54:03,657 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-15 02:54:03,657 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-15 02:54:03,657 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:03,657 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-15 02:54:03,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-15 02:54:03,657 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:03,657 DEBUG [Thread-864] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-15 02:54:03,657 DEBUG [Thread-864] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-15 02:54:03,657 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:03,757 DEBUG [Thread-864] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-15 02:54:03,758 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-15 02:54:03,760 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-15 02:54:03,770 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-15 02:54:03,770 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-15 02:54:03,770 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-15 02:54:03,770 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-15 02:54:03,770 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-15 02:54:03,771 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-15 02:54:03,771 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:03,771 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:54:03,772 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:03,772 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-15 02:54:03,772 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-15 02:54:03,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-15 02:54:03,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:54:03,772 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:03,773 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-15 02:54:03,773 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:03,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-15 02:54:03,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:54:03,773 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-15 02:54:03,774 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b7b06ed62cbe8eca9103c3d111564e6c, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:54:03,775 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34681] ipc.CallRunner(144): callId: 11 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:39106 deadline: 1681527303774, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b7b06ed62cbe8eca9103c3d111564e6c, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:03,786 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-15 02:54:03,786 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-15 02:54:03,786 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-15 02:54:03,786 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-15 02:54:03,786 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-15 02:54:03,787 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b7b06ed62cbe8eca9103c3d111564e6c, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:54:03,787 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b7b06ed62cbe8eca9103c3d111564e6c, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:54:03,787 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34681] ipc.CallRunner(144): callId: 16 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:39108 deadline: 1681527303785, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b7b06ed62cbe8eca9103c3d111564e6c, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:03,787 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34681] ipc.CallRunner(144): callId: 15 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:39132 deadline: 1681527303785, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b7b06ed62cbe8eca9103c3d111564e6c, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:03,787 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b7b06ed62cbe8eca9103c3d111564e6c, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:54:03,787 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34681] ipc.CallRunner(144): callId: 15 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:39116 deadline: 1681527303786, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b7b06ed62cbe8eca9103c3d111564e6c, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:03,787 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b7b06ed62cbe8eca9103c3d111564e6c, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:54:03,789 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34681] ipc.CallRunner(144): callId: 13 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:39142 deadline: 1681527303787, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b7b06ed62cbe8eca9103c3d111564e6c, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:03,798 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-15 02:54:03,798 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-15 02:54:03,798 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681527240182.b7b06ed62cbe8eca9103c3d111564e6c. 2023-04-15 02:54:03,798 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681527240182.b7b06ed62cbe8eca9103c3d111564e6c. started... 2023-04-15 02:54:03,798 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681527240182.b7b06ed62cbe8eca9103c3d111564e6c. as already flushing 2023-04-15 02:54:03,798 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681527240182.b7b06ed62cbe8eca9103c3d111564e6c. 2023-04-15 02:54:03,798 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-15 02:54:03,798 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-15 02:54:03,798 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:03,798 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-15 02:54:03,798 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34681,1681527209133' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-15 02:54:03,799 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:03,799 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:03,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:03,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:54:03,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:54:03,800 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:03,800 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-15 02:54:03,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:54:03,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:54:03,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:54:03,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:03,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:54:03,801 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34681,1681527209133' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-15 02:54:03,802 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@21e6bd69[Count = 0] remaining members to acquire global barrier 2023-04-15 02:54:03,802 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-15 02:54:03,802 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:03,803 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:03,803 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:03,803 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-15 02:54:03,803 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:03,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:03,803 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-15 02:54:03,803 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-15 02:54:03,803 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34681,1681527209133' in zk 2023-04-15 02:54:03,804 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:03,804 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-15 02:54:03,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:03,804 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-15 02:54:03,804 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-15 02:54:03,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:54:03,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:54:03,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:54:03,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:54:03,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:54:03,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:03,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:54:03,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:54:03,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:03,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34681,1681527209133': 2023-04-15 02:54:03,808 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34681,1681527209133' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-15 02:54:03,808 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-15 02:54:03,808 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-15 02:54:03,808 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-15 02:54:03,808 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-15 02:54:03,808 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-15 02:54:03,809 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:03,809 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:03,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:03,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:54:03,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:54:03,810 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:03,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:03,810 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-15 02:54:03,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:54:03,810 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-15 02:54:03,810 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:03,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-15 02:54:03,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:54:03,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:03,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:54:03,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:54:03,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:03,812 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:03,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:54:03,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:54:03,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:03,814 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:03,814 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-15 02:54:03,814 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-15 02:54:03,814 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-15 02:54:03,814 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:54:03,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-15 02:54:03,815 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-15 02:54:03,815 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-15 02:54:03,814 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:54:03,814 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-15 02:54:03,815 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:03,815 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:03,815 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:03,815 DEBUG [Thread-864] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-15 02:54:03,815 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:03,815 DEBUG [Thread-864] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-15 02:54:03,815 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-15 02:54:03,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-15 02:54:03,852 WARN [HBase-Metrics2-1] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties 2023-04-15 02:54:03,915 DEBUG [Thread-864] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-15 02:54:03,916 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-15 02:54:03,917 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-15 02:54:03,919 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-15 02:54:03,919 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-15 02:54:03,919 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-15 02:54:03,919 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-15 02:54:03,919 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-15 02:54:03,919 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-15 02:54:03,919 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:03,919 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:54:03,920 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-15 02:54:03,920 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:03,920 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-15 02:54:03,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-15 02:54:03,921 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:03,921 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-15 02:54:03,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:54:03,921 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:03,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-15 02:54:03,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:54:03,921 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-15 02:54:03,921 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-15 02:54:03,921 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-15 02:54:03,922 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-15 02:54:03,922 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-15 02:54:03,922 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-15 02:54:03,922 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-15 02:54:03,922 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-15 02:54:03,922 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681527240182.b7b06ed62cbe8eca9103c3d111564e6c. 2023-04-15 02:54:03,922 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681527240182.b7b06ed62cbe8eca9103c3d111564e6c. started... 2023-04-15 02:54:03,922 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681527240182.b7b06ed62cbe8eca9103c3d111564e6c. as already flushing 2023-04-15 02:54:03,922 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681527240182.b7b06ed62cbe8eca9103c3d111564e6c. 2023-04-15 02:54:03,922 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-15 02:54:03,922 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-15 02:54:03,922 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:03,922 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-15 02:54:03,922 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34681,1681527209133' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-15 02:54:03,923 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:03,923 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:03,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:03,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:54:03,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:54:03,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:54:03,924 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:03,924 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-15 02:54:03,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:54:03,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:54:03,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:03,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:54:03,925 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34681,1681527209133' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-15 02:54:03,925 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-15 02:54:03,925 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@31602dab[Count = 0] remaining members to acquire global barrier 2023-04-15 02:54:03,925 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:03,926 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:03,926 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:03,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:03,926 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-15 02:54:03,926 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:03,926 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-15 02:54:03,926 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-15 02:54:03,926 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34681,1681527209133' in zk 2023-04-15 02:54:03,927 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-15 02:54:03,927 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:03,927 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-15 02:54:03,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:03,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:54:03,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:54:03,927 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-15 02:54:03,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:54:03,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:54:03,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:54:03,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:03,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:54:03,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:54:03,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:03,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34681,1681527209133': 2023-04-15 02:54:03,930 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34681,1681527209133' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-15 02:54:03,930 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-15 02:54:03,930 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-15 02:54:03,930 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-15 02:54:03,930 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-15 02:54:03,930 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-15 02:54:03,931 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:03,931 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:03,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:03,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:54:03,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:54:03,931 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:03,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:03,931 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-15 02:54:03,931 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-15 02:54:03,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-15 02:54:03,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:03,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:54:03,932 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:03,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:54:03,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:54:03,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:54:03,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:03,933 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:03,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:54:03,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:54:03,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:03,935 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-15 02:54:03,935 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-15 02:54:03,935 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-15 02:54:03,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-15 02:54:03,935 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:03,935 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-15 02:54:03,935 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-15 02:54:03,935 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-15 02:54:03,935 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-15 02:54:03,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-15 02:54:03,935 DEBUG [Thread-864] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-15 02:54:03,935 DEBUG [Thread-864] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-15 02:54:03,935 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:54:03,935 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:54:03,936 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:03,936 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:03,936 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:03,936 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:04,016 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=16 (bloomFilter=true), to=hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/b7b06ed62cbe8eca9103c3d111564e6c/.tmp/C/353790da0a4f4819b1625c8eb352666e 2023-04-15 02:54:04,021 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/b7b06ed62cbe8eca9103c3d111564e6c/.tmp/A/fc91b7d3afe3454099e64a7900c7f638 as hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/b7b06ed62cbe8eca9103c3d111564e6c/A/fc91b7d3afe3454099e64a7900c7f638 2023-04-15 02:54:04,026 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/b7b06ed62cbe8eca9103c3d111564e6c/A/fc91b7d3afe3454099e64a7900c7f638, entries=200, sequenceid=16, filesize=31.9 K 2023-04-15 02:54:04,027 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/b7b06ed62cbe8eca9103c3d111564e6c/.tmp/B/1898232003dd448791f6d6b222731e5a as hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/b7b06ed62cbe8eca9103c3d111564e6c/B/1898232003dd448791f6d6b222731e5a 2023-04-15 02:54:04,032 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/b7b06ed62cbe8eca9103c3d111564e6c/B/1898232003dd448791f6d6b222731e5a, entries=150, sequenceid=16, filesize=11.6 K 2023-04-15 02:54:04,033 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/b7b06ed62cbe8eca9103c3d111564e6c/.tmp/C/353790da0a4f4819b1625c8eb352666e as hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/b7b06ed62cbe8eca9103c3d111564e6c/C/353790da0a4f4819b1625c8eb352666e 2023-04-15 02:54:04,036 DEBUG [Thread-864] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-15 02:54:04,038 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-15 02:54:04,043 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-15 02:54:04,045 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/b7b06ed62cbe8eca9103c3d111564e6c/C/353790da0a4f4819b1625c8eb352666e, entries=150, sequenceid=16, filesize=11.6 K 2023-04-15 02:54:04,046 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~60.38 KB/61830, heapSize ~180 KB/184320, currentSize=134.18 KB/137400 for b7b06ed62cbe8eca9103c3d111564e6c in 611ms, sequenceid=16, compaction requested=false 2023-04-15 02:54:04,046 DEBUG [MemStoreFlusher.0] regionserver.MetricsTableSourceImpl(130): Creating new MetricsTableSourceImpl for table 'TestAcidGuarantees' 2023-04-15 02:54:04,047 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for b7b06ed62cbe8eca9103c3d111564e6c: 2023-04-15 02:54:04,053 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-15 02:54:04,053 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-15 02:54:04,053 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-15 02:54:04,053 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-15 02:54:04,053 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-15 02:54:04,053 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-15 02:54:04,054 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:04,054 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:54:04,054 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:04,054 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-15 02:54:04,055 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-15 02:54:04,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-15 02:54:04,055 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:04,055 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-15 02:54:04,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:54:04,056 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:04,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-15 02:54:04,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:54:04,056 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-15 02:54:04,057 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-15 02:54:04,057 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-15 02:54:04,057 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-15 02:54:04,057 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-15 02:54:04,057 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-15 02:54:04,066 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-15 02:54:04,066 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681527240182.b7b06ed62cbe8eca9103c3d111564e6c. 2023-04-15 02:54:04,066 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-15 02:54:04,066 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681527240182.b7b06ed62cbe8eca9103c3d111564e6c. started... 2023-04-15 02:54:04,066 INFO [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing b7b06ed62cbe8eca9103c3d111564e6c 3/3 column families, dataSize=134.18 KB heapSize=370.92 KB 2023-04-15 02:54:04,066 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK b7b06ed62cbe8eca9103c3d111564e6c, store=A 2023-04-15 02:54:04,067 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-15 02:54:04,067 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK b7b06ed62cbe8eca9103c3d111564e6c, store=B 2023-04-15 02:54:04,067 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-15 02:54:04,067 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK b7b06ed62cbe8eca9103c3d111564e6c, store=C 2023-04-15 02:54:04,067 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-15 02:54:04,079 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34681] regionserver.HRegion(9158): Flush requested on b7b06ed62cbe8eca9103c3d111564e6c 2023-04-15 02:54:04,079 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681527240182.b7b06ed62cbe8eca9103c3d111564e6c. as already flushing 2023-04-15 02:54:04,096 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b7b06ed62cbe8eca9103c3d111564e6c, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:54:04,096 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34681] ipc.CallRunner(144): callId: 18 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:39108 deadline: 1681527304089, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b7b06ed62cbe8eca9103c3d111564e6c, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:04,097 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b7b06ed62cbe8eca9103c3d111564e6c, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:54:04,097 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34681] ipc.CallRunner(144): callId: 15 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:39142 deadline: 1681527304090, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b7b06ed62cbe8eca9103c3d111564e6c, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:04,097 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b7b06ed62cbe8eca9103c3d111564e6c, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:54:04,097 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34681] ipc.CallRunner(144): callId: 17 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:39116 deadline: 1681527304090, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b7b06ed62cbe8eca9103c3d111564e6c, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:04,098 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b7b06ed62cbe8eca9103c3d111564e6c, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:54:04,098 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34681] ipc.CallRunner(144): callId: 20 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:39106 deadline: 1681527304095, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b7b06ed62cbe8eca9103c3d111564e6c, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:04,098 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b7b06ed62cbe8eca9103c3d111564e6c, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:54:04,098 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34681] ipc.CallRunner(144): callId: 18 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:39132 deadline: 1681527304096, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b7b06ed62cbe8eca9103c3d111564e6c, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:04,120 INFO [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230415b9a16768821f4f3ebc37383f80930a9e to hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230415b9a16768821f4f3ebc37383f80930a9e 2023-04-15 02:54:04,134 INFO [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=50, memsize=44.7 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/b7b06ed62cbe8eca9103c3d111564e6c/.tmp/A/8f538e98aa65491ebfab8d01af624c27 2023-04-15 02:54:04,168 INFO [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=50 (bloomFilter=true), to=hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/b7b06ed62cbe8eca9103c3d111564e6c/.tmp/B/91193976fe4c4e15b19485e66f2f6950 2023-04-15 02:54:04,200 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b7b06ed62cbe8eca9103c3d111564e6c, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:54:04,201 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34681] ipc.CallRunner(144): callId: 20 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:39132 deadline: 1681527304199, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b7b06ed62cbe8eca9103c3d111564e6c, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:04,201 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b7b06ed62cbe8eca9103c3d111564e6c, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:54:04,201 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34681] ipc.CallRunner(144): callId: 22 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:39106 deadline: 1681527304199, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b7b06ed62cbe8eca9103c3d111564e6c, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:04,403 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b7b06ed62cbe8eca9103c3d111564e6c, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:54:04,403 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34681] ipc.CallRunner(144): callId: 22 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:39132 deadline: 1681527304402, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b7b06ed62cbe8eca9103c3d111564e6c, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:04,403 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b7b06ed62cbe8eca9103c3d111564e6c, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:54:04,403 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34681] ipc.CallRunner(144): callId: 24 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:39106 deadline: 1681527304402, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b7b06ed62cbe8eca9103c3d111564e6c, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:04,594 INFO [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=50 (bloomFilter=true), to=hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/b7b06ed62cbe8eca9103c3d111564e6c/.tmp/C/deb3773513d94fcf990a9633a3cfe8f2 2023-04-15 02:54:04,599 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/b7b06ed62cbe8eca9103c3d111564e6c/.tmp/A/8f538e98aa65491ebfab8d01af624c27 as hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/b7b06ed62cbe8eca9103c3d111564e6c/A/8f538e98aa65491ebfab8d01af624c27 2023-04-15 02:54:04,600 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b7b06ed62cbe8eca9103c3d111564e6c, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:54:04,600 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34681] ipc.CallRunner(144): callId: 19 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:39116 deadline: 1681527304599, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b7b06ed62cbe8eca9103c3d111564e6c, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:04,600 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b7b06ed62cbe8eca9103c3d111564e6c, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:54:04,600 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34681] ipc.CallRunner(144): callId: 20 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:39108 deadline: 1681527304600, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b7b06ed62cbe8eca9103c3d111564e6c, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:04,603 INFO [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/b7b06ed62cbe8eca9103c3d111564e6c/A/8f538e98aa65491ebfab8d01af624c27, entries=150, sequenceid=50, filesize=25.1 K 2023-04-15 02:54:04,604 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b7b06ed62cbe8eca9103c3d111564e6c, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:54:04,604 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34681] ipc.CallRunner(144): callId: 17 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:39142 deadline: 1681527304601, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b7b06ed62cbe8eca9103c3d111564e6c, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:04,605 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/b7b06ed62cbe8eca9103c3d111564e6c/.tmp/B/91193976fe4c4e15b19485e66f2f6950 as hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/b7b06ed62cbe8eca9103c3d111564e6c/B/91193976fe4c4e15b19485e66f2f6950 2023-04-15 02:54:04,609 INFO [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/b7b06ed62cbe8eca9103c3d111564e6c/B/91193976fe4c4e15b19485e66f2f6950, entries=150, sequenceid=50, filesize=11.6 K 2023-04-15 02:54:04,610 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/b7b06ed62cbe8eca9103c3d111564e6c/.tmp/C/deb3773513d94fcf990a9633a3cfe8f2 as hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/b7b06ed62cbe8eca9103c3d111564e6c/C/deb3773513d94fcf990a9633a3cfe8f2 2023-04-15 02:54:04,615 INFO [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/b7b06ed62cbe8eca9103c3d111564e6c/C/deb3773513d94fcf990a9633a3cfe8f2, entries=150, sequenceid=50, filesize=11.6 K 2023-04-15 02:54:04,616 INFO [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~134.18 KB/137400, heapSize ~370.88 KB/379776, currentSize=53.67 KB/54960 for b7b06ed62cbe8eca9103c3d111564e6c in 550ms, sequenceid=50, compaction requested=false 2023-04-15 02:54:04,617 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for b7b06ed62cbe8eca9103c3d111564e6c: 2023-04-15 02:54:04,617 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681527240182.b7b06ed62cbe8eca9103c3d111564e6c. 2023-04-15 02:54:04,617 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-15 02:54:04,617 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-15 02:54:04,617 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:04,617 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-15 02:54:04,617 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34681,1681527209133' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-15 02:54:04,618 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:04,618 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:04,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:04,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:54:04,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:54:04,619 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:04,619 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-15 02:54:04,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:54:04,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:54:04,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:54:04,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:04,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:54:04,620 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34681,1681527209133' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-15 02:54:04,620 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-15 02:54:04,620 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1b820b81[Count = 0] remaining members to acquire global barrier 2023-04-15 02:54:04,620 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:04,620 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:04,621 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:04,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:04,621 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-15 02:54:04,621 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-15 02:54:04,621 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:04,621 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-15 02:54:04,621 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34681,1681527209133' in zk 2023-04-15 02:54:04,622 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:04,622 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-15 02:54:04,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:04,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:54:04,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:54:04,622 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-15 02:54:04,622 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-15 02:54:04,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:54:04,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:54:04,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:54:04,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:04,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:54:04,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:54:04,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:04,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34681,1681527209133': 2023-04-15 02:54:04,624 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34681,1681527209133' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-15 02:54:04,624 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-15 02:54:04,624 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-15 02:54:04,624 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-15 02:54:04,624 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-15 02:54:04,624 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-15 02:54:04,625 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:04,625 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-15 02:54:04,625 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:04,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:04,625 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:04,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:04,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:54:04,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:54:04,626 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-15 02:54:04,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-15 02:54:04,626 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:04,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:54:04,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:04,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:54:04,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:54:04,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:54:04,628 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:04,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:04,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:54:04,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:54:04,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:04,630 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:04,630 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:54:04,630 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:54:04,630 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:04,630 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:04,630 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-15 02:54:04,630 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-15 02:54:04,630 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-15 02:54:04,630 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:04,630 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:04,630 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-15 02:54:04,630 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-15 02:54:04,630 DEBUG [Thread-864] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-15 02:54:04,630 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-15 02:54:04,630 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-15 02:54:04,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-15 02:54:04,630 DEBUG [Thread-864] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-15 02:54:04,631 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-15 02:54:04,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-15 02:54:04,707 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34681] regionserver.HRegion(9158): Flush requested on b7b06ed62cbe8eca9103c3d111564e6c 2023-04-15 02:54:04,707 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing b7b06ed62cbe8eca9103c3d111564e6c 3/3 column families, dataSize=60.38 KB heapSize=180.05 KB 2023-04-15 02:54:04,707 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK b7b06ed62cbe8eca9103c3d111564e6c, store=A 2023-04-15 02:54:04,707 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-15 02:54:04,707 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK b7b06ed62cbe8eca9103c3d111564e6c, store=B 2023-04-15 02:54:04,707 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-15 02:54:04,707 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK b7b06ed62cbe8eca9103c3d111564e6c, store=C 2023-04-15 02:54:04,707 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-15 02:54:04,726 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230415c50d7e5025b7481da1600302c3db1bae to hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230415c50d7e5025b7481da1600302c3db1bae 2023-04-15 02:54:04,731 DEBUG [Thread-864] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-15 02:54:04,732 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-15 02:54:04,739 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-15 02:54:04,739 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=62, memsize=20.1 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/b7b06ed62cbe8eca9103c3d111564e6c/.tmp/A/7219c23fbaee4d32b1f84582e92030c0 2023-04-15 02:54:04,751 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1681527244738, max timestamp=1681527244747Num uniques 859; 2023-04-15 02:54:04,751 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1681527244738, max timestamp=1681527244747Num uniques 859; 2023-04-15 02:54:04,751 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-7] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-15 02:54:04,751 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-4] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-15 02:54:04,751 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-15 02:54:04,751 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-15 02:54:04,751 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-15 02:54:04,752 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-15 02:54:04,752 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-15 02:54:04,752 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-15 02:54:04,752 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:04,752 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:54:04,753 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681527244743, max timestamp=1681527244747Num uniques 859; 2023-04-15 02:54:04,754 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-9] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-15 02:54:04,754 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-15 02:54:04,754 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:04,754 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-15 02:54:04,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-15 02:54:04,754 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:04,755 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-15 02:54:04,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:54:04,755 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:04,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-15 02:54:04,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:54:04,755 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-15 02:54:04,758 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-15 02:54:04,758 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-15 02:54:04,758 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-15 02:54:04,758 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-15 02:54:04,758 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-15 02:54:04,770 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-15 02:54:04,770 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-15 02:54:04,770 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681527240182.b7b06ed62cbe8eca9103c3d111564e6c. 2023-04-15 02:54:04,770 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681527240182.b7b06ed62cbe8eca9103c3d111564e6c. started... 2023-04-15 02:54:04,770 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681527240182.b7b06ed62cbe8eca9103c3d111564e6c. as already flushing 2023-04-15 02:54:04,770 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681527240182.b7b06ed62cbe8eca9103c3d111564e6c. 2023-04-15 02:54:04,770 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-15 02:54:04,770 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-15 02:54:04,770 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:04,771 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-15 02:54:04,771 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34681,1681527209133' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-15 02:54:04,772 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b7b06ed62cbe8eca9103c3d111564e6c, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:54:04,772 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34681] ipc.CallRunner(144): callId: 42 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:39106 deadline: 1681527304771, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b7b06ed62cbe8eca9103c3d111564e6c, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:04,772 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:04,772 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:04,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:04,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:54:04,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:54:04,773 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:04,773 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-15 02:54:04,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:54:04,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:54:04,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:54:04,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:04,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:54:04,774 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34681,1681527209133' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-15 02:54:04,774 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@74d808c0[Count = 0] remaining members to acquire global barrier 2023-04-15 02:54:04,774 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-15 02:54:04,774 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:04,775 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b7b06ed62cbe8eca9103c3d111564e6c, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:54:04,775 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34681] ipc.CallRunner(144): callId: 40 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:39132 deadline: 1681527304773, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b7b06ed62cbe8eca9103c3d111564e6c, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:04,775 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:04,775 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:04,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:04,775 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-15 02:54:04,775 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-15 02:54:04,775 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34681,1681527209133' in zk 2023-04-15 02:54:04,776 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:04,776 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-15 02:54:04,776 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-15 02:54:04,776 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:04,776 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-15 02:54:04,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:04,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:54:04,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:54:04,776 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-15 02:54:04,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:54:04,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:54:04,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:54:04,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:04,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:54:04,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:54:04,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:04,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34681,1681527209133': 2023-04-15 02:54:04,779 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34681,1681527209133' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-15 02:54:04,779 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-15 02:54:04,779 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-15 02:54:04,779 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-15 02:54:04,779 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-15 02:54:04,779 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-15 02:54:04,780 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:04,780 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-15 02:54:04,780 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:04,780 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:04,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:04,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:04,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:54:04,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:54:04,781 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-15 02:54:04,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-15 02:54:04,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:04,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:54:04,781 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:04,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:54:04,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:54:04,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:54:04,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:04,783 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:04,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:54:04,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:54:04,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:04,793 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:04,793 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:54:04,794 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:54:04,794 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-15 02:54:04,794 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-15 02:54:04,794 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-15 02:54:04,794 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-15 02:54:04,794 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-15 02:54:04,794 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:04,794 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-15 02:54:04,794 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:04,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-15 02:54:04,794 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:04,794 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:04,794 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-15 02:54:04,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-15 02:54:04,794 DEBUG [Thread-864] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-15 02:54:04,795 DEBUG [Thread-864] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-15 02:54:04,874 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b7b06ed62cbe8eca9103c3d111564e6c, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:54:04,874 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34681] ipc.CallRunner(144): callId: 44 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:39106 deadline: 1681527304873, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b7b06ed62cbe8eca9103c3d111564e6c, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:04,877 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b7b06ed62cbe8eca9103c3d111564e6c, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:54:04,877 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34681] ipc.CallRunner(144): callId: 42 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:39132 deadline: 1681527304876, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b7b06ed62cbe8eca9103c3d111564e6c, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:04,895 DEBUG [Thread-864] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-15 02:54:04,895 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-15 02:54:04,897 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-15 02:54:04,898 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-15 02:54:04,898 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-15 02:54:04,898 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-15 02:54:04,898 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-15 02:54:04,898 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-15 02:54:04,898 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-15 02:54:04,899 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:04,899 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:54:04,899 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:04,899 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-15 02:54:04,899 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-15 02:54:04,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-15 02:54:04,899 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:04,899 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-15 02:54:04,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:54:04,900 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:04,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-15 02:54:04,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:54:04,900 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-15 02:54:04,900 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-15 02:54:04,901 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-15 02:54:04,901 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-15 02:54:04,901 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-15 02:54:04,901 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-15 02:54:04,901 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-15 02:54:04,901 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-15 02:54:04,901 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681527240182.b7b06ed62cbe8eca9103c3d111564e6c. 2023-04-15 02:54:04,901 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681527240182.b7b06ed62cbe8eca9103c3d111564e6c. started... 2023-04-15 02:54:04,901 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681527240182.b7b06ed62cbe8eca9103c3d111564e6c. as already flushing 2023-04-15 02:54:04,901 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681527240182.b7b06ed62cbe8eca9103c3d111564e6c. 2023-04-15 02:54:04,901 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-15 02:54:04,901 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-15 02:54:04,901 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:04,901 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-15 02:54:04,901 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34681,1681527209133' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-15 02:54:04,902 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:04,902 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:04,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:04,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:54:04,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:54:04,902 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:04,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:54:04,902 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-15 02:54:04,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:54:04,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:54:04,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:04,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:54:04,904 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34681,1681527209133' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-15 02:54:04,904 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6041eec7[Count = 0] remaining members to acquire global barrier 2023-04-15 02:54:04,904 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-15 02:54:04,904 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:04,904 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:04,904 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:04,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:04,904 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-15 02:54:04,904 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:04,904 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-15 02:54:04,904 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-15 02:54:04,904 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34681,1681527209133' in zk 2023-04-15 02:54:04,906 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:04,906 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-15 02:54:04,906 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-15 02:54:04,906 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-15 02:54:04,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:04,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:54:04,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:54:04,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:54:04,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:54:04,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:54:04,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:04,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:54:04,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:54:04,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:04,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34681,1681527209133': 2023-04-15 02:54:04,907 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34681,1681527209133' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-15 02:54:04,907 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-15 02:54:04,908 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-15 02:54:04,908 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-15 02:54:04,908 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-15 02:54:04,908 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-15 02:54:04,908 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:04,908 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:04,908 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:04,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:04,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:54:04,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:54:04,908 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-15 02:54:04,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:04,909 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:04,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:54:04,909 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-15 02:54:04,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-15 02:54:04,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:04,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:54:04,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:54:04,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:54:04,910 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:04,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:04,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:54:04,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:54:04,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:04,911 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:04,911 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:54:04,911 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-15 02:54:04,911 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:54:04,911 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-15 02:54:04,911 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-15 02:54:04,911 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-15 02:54:04,911 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:04,911 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-15 02:54:04,911 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-15 02:54:04,911 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:04,911 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-15 02:54:04,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-15 02:54:04,911 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:04,911 DEBUG [Thread-864] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-15 02:54:04,912 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:04,912 DEBUG [Thread-864] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-15 02:54:04,912 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-15 02:54:04,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-15 02:54:05,012 DEBUG [Thread-864] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-15 02:54:05,012 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-15 02:54:05,014 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-15 02:54:05,027 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-15 02:54:05,027 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-15 02:54:05,027 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-15 02:54:05,027 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-15 02:54:05,027 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-15 02:54:05,027 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-15 02:54:05,028 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:05,028 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:54:05,028 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:05,028 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-15 02:54:05,028 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-15 02:54:05,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-15 02:54:05,029 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:05,029 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-15 02:54:05,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:54:05,029 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:05,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-15 02:54:05,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:54:05,030 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-15 02:54:05,030 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-15 02:54:05,030 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-15 02:54:05,030 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-15 02:54:05,030 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-15 02:54:05,030 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-15 02:54:05,030 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-15 02:54:05,030 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-15 02:54:05,030 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681527240182.b7b06ed62cbe8eca9103c3d111564e6c. 2023-04-15 02:54:05,030 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681527240182.b7b06ed62cbe8eca9103c3d111564e6c. started... 2023-04-15 02:54:05,031 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681527240182.b7b06ed62cbe8eca9103c3d111564e6c. as already flushing 2023-04-15 02:54:05,031 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681527240182.b7b06ed62cbe8eca9103c3d111564e6c. 2023-04-15 02:54:05,031 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-15 02:54:05,031 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-15 02:54:05,031 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:05,031 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-15 02:54:05,031 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34681,1681527209133' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-15 02:54:05,032 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:05,032 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:05,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:05,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:54:05,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:54:05,032 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:05,032 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-15 02:54:05,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:54:05,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:54:05,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:54:05,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:05,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:54:05,033 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34681,1681527209133' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-15 02:54:05,033 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@507c9a11[Count = 0] remaining members to acquire global barrier 2023-04-15 02:54:05,033 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-15 02:54:05,033 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:05,033 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:05,033 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:05,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:05,034 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:05,034 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-15 02:54:05,034 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-15 02:54:05,034 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-15 02:54:05,034 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34681,1681527209133' in zk 2023-04-15 02:54:05,034 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:05,034 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-15 02:54:05,034 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-15 02:54:05,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:05,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:54:05,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:54:05,034 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-15 02:54:05,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:54:05,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:54:05,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:54:05,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:05,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:54:05,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:54:05,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:05,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34681,1681527209133': 2023-04-15 02:54:05,036 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34681,1681527209133' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-15 02:54:05,036 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-15 02:54:05,036 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-15 02:54:05,036 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-15 02:54:05,036 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-15 02:54:05,036 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-15 02:54:05,037 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:05,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:05,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:54:05,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:54:05,037 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:05,037 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-15 02:54:05,037 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:05,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:05,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:54:05,037 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-15 02:54:05,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-15 02:54:05,037 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:05,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:54:05,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:05,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:54:05,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:54:05,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:05,038 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:05,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:54:05,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:54:05,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:05,039 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:05,040 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:54:05,040 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:54:05,040 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:05,040 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:05,040 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-15 02:54:05,040 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-15 02:54:05,040 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-15 02:54:05,040 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-15 02:54:05,040 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-15 02:54:05,040 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-15 02:54:05,040 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:05,040 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:05,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-15 02:54:05,040 DEBUG [Thread-864] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-15 02:54:05,040 DEBUG [Thread-864] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-15 02:54:05,040 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-15 02:54:05,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-15 02:54:05,077 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b7b06ed62cbe8eca9103c3d111564e6c, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:54:05,077 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34681] ipc.CallRunner(144): callId: 46 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:39106 deadline: 1681527305076, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b7b06ed62cbe8eca9103c3d111564e6c, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:05,081 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b7b06ed62cbe8eca9103c3d111564e6c, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:54:05,081 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34681] ipc.CallRunner(144): callId: 44 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:39132 deadline: 1681527305079, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b7b06ed62cbe8eca9103c3d111564e6c, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:05,140 DEBUG [Thread-864] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-15 02:54:05,141 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-15 02:54:05,142 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-15 02:54:05,143 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-15 02:54:05,143 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-15 02:54:05,143 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-15 02:54:05,144 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-15 02:54:05,144 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-15 02:54:05,144 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-15 02:54:05,144 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:05,144 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:54:05,145 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:05,145 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-15 02:54:05,145 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-15 02:54:05,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-15 02:54:05,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:54:05,145 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:05,145 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-15 02:54:05,145 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:05,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-15 02:54:05,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:54:05,146 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-15 02:54:05,146 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-15 02:54:05,146 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-15 02:54:05,146 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-15 02:54:05,146 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-15 02:54:05,146 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-15 02:54:05,148 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-15 02:54:05,148 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-15 02:54:05,148 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681527240182.b7b06ed62cbe8eca9103c3d111564e6c. 2023-04-15 02:54:05,148 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681527240182.b7b06ed62cbe8eca9103c3d111564e6c. started... 2023-04-15 02:54:05,148 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681527240182.b7b06ed62cbe8eca9103c3d111564e6c. as already flushing 2023-04-15 02:54:05,148 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681527240182.b7b06ed62cbe8eca9103c3d111564e6c. 2023-04-15 02:54:05,148 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-15 02:54:05,148 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-15 02:54:05,148 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:05,148 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-15 02:54:05,148 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34681,1681527209133' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-15 02:54:05,149 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:05,150 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:05,150 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:05,150 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-15 02:54:05,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:05,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:54:05,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:54:05,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:54:05,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:54:05,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:54:05,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:05,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:54:05,152 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34681,1681527209133' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-15 02:54:05,152 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-15 02:54:05,152 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5b294d9c[Count = 0] remaining members to acquire global barrier 2023-04-15 02:54:05,152 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:05,152 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:05,152 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:05,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:05,152 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-15 02:54:05,152 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-15 02:54:05,152 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34681,1681527209133' in zk 2023-04-15 02:54:05,153 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:05,153 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-15 02:54:05,153 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:05,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:05,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:54:05,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:54:05,153 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-15 02:54:05,154 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-15 02:54:05,154 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-15 02:54:05,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:54:05,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:54:05,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:54:05,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:05,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:54:05,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:54:05,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:05,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34681,1681527209133': 2023-04-15 02:54:05,156 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34681,1681527209133' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-15 02:54:05,156 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-15 02:54:05,156 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-15 02:54:05,156 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-15 02:54:05,156 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-15 02:54:05,156 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-15 02:54:05,157 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:05,157 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:05,157 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-15 02:54:05,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:05,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:54:05,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:54:05,157 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:05,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:05,158 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:05,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:54:05,158 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-15 02:54:05,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-15 02:54:05,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:54:05,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:05,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:54:05,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:54:05,159 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:05,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:05,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:54:05,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:54:05,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:05,160 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:05,160 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:54:05,160 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:54:05,160 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:05,160 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:05,160 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-15 02:54:05,160 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-15 02:54:05,160 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-15 02:54:05,160 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-15 02:54:05,160 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-15 02:54:05,160 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:05,160 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-15 02:54:05,160 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-15 02:54:05,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-15 02:54:05,160 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:05,161 DEBUG [Thread-864] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-15 02:54:05,161 DEBUG [Thread-864] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-15 02:54:05,161 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-15 02:54:05,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-15 02:54:05,185 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=62 (bloomFilter=true), to=hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/b7b06ed62cbe8eca9103c3d111564e6c/.tmp/B/8044099aed62448096cf6b59026c51f9 2023-04-15 02:54:05,201 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=62 (bloomFilter=true), to=hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/b7b06ed62cbe8eca9103c3d111564e6c/.tmp/C/38eaea4cbf6e46cbbe3107d8924ce952 2023-04-15 02:54:05,206 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/b7b06ed62cbe8eca9103c3d111564e6c/.tmp/A/7219c23fbaee4d32b1f84582e92030c0 as hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/b7b06ed62cbe8eca9103c3d111564e6c/A/7219c23fbaee4d32b1f84582e92030c0 2023-04-15 02:54:05,210 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/b7b06ed62cbe8eca9103c3d111564e6c/A/7219c23fbaee4d32b1f84582e92030c0, entries=200, sequenceid=62, filesize=31.9 K 2023-04-15 02:54:05,211 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/b7b06ed62cbe8eca9103c3d111564e6c/.tmp/B/8044099aed62448096cf6b59026c51f9 as hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/b7b06ed62cbe8eca9103c3d111564e6c/B/8044099aed62448096cf6b59026c51f9 2023-04-15 02:54:05,215 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/b7b06ed62cbe8eca9103c3d111564e6c/B/8044099aed62448096cf6b59026c51f9, entries=150, sequenceid=62, filesize=11.6 K 2023-04-15 02:54:05,216 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/b7b06ed62cbe8eca9103c3d111564e6c/.tmp/C/38eaea4cbf6e46cbbe3107d8924ce952 as hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/b7b06ed62cbe8eca9103c3d111564e6c/C/38eaea4cbf6e46cbbe3107d8924ce952 2023-04-15 02:54:05,220 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/b7b06ed62cbe8eca9103c3d111564e6c/C/38eaea4cbf6e46cbbe3107d8924ce952, entries=150, sequenceid=62, filesize=11.6 K 2023-04-15 02:54:05,221 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~60.38 KB/61830, heapSize ~180 KB/184320, currentSize=118.53 KB/121370 for b7b06ed62cbe8eca9103c3d111564e6c in 514ms, sequenceid=62, compaction requested=true 2023-04-15 02:54:05,221 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for b7b06ed62cbe8eca9103c3d111564e6c: 2023-04-15 02:54:05,221 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-15 02:54:05,221 DEBUG [RS:0;jenkins-hbase20:34681-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-15 02:54:05,221 DEBUG [RS:0;jenkins-hbase20:34681-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-15 02:54:05,221 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-15 02:54:05,222 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-15 02:54:05,223 DEBUG [RS:0;jenkins-hbase20:34681-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 91118 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-15 02:54:05,223 DEBUG [RS:0;jenkins-hbase20:34681-shortCompactions-0] regionserver.HStore(1912): b7b06ed62cbe8eca9103c3d111564e6c/A is initiating minor compaction (all files) 2023-04-15 02:54:05,223 INFO [RS:0;jenkins-hbase20:34681-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of b7b06ed62cbe8eca9103c3d111564e6c/A in TestAcidGuarantees,,1681527240182.b7b06ed62cbe8eca9103c3d111564e6c. 2023-04-15 02:54:05,223 INFO [RS:0;jenkins-hbase20:34681-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/b7b06ed62cbe8eca9103c3d111564e6c/A/fc91b7d3afe3454099e64a7900c7f638, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/b7b06ed62cbe8eca9103c3d111564e6c/A/8f538e98aa65491ebfab8d01af624c27, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/b7b06ed62cbe8eca9103c3d111564e6c/A/7219c23fbaee4d32b1f84582e92030c0] into tmpdir=hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/b7b06ed62cbe8eca9103c3d111564e6c/.tmp, totalSize=89.0 K 2023-04-15 02:54:05,223 DEBUG [RS:0;jenkins-hbase20:34681-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 35658 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-15 02:54:05,224 DEBUG [RS:0;jenkins-hbase20:34681-shortCompactions-0] compactions.Compactor(207): Compacting fc91b7d3afe3454099e64a7900c7f638, keycount=200, bloomtype=ROW, size=31.9 K, encoding=NONE, compression=NONE, seqNum=16, earliestPutTs=1681527243418 2023-04-15 02:54:05,224 DEBUG [RS:0;jenkins-hbase20:34681-longCompactions-0] regionserver.HStore(1912): b7b06ed62cbe8eca9103c3d111564e6c/B is initiating minor compaction (all files) 2023-04-15 02:54:05,224 INFO [RS:0;jenkins-hbase20:34681-longCompactions-0] regionserver.HRegion(2259): Starting compaction of b7b06ed62cbe8eca9103c3d111564e6c/B in TestAcidGuarantees,,1681527240182.b7b06ed62cbe8eca9103c3d111564e6c. 2023-04-15 02:54:05,224 INFO [RS:0;jenkins-hbase20:34681-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/b7b06ed62cbe8eca9103c3d111564e6c/B/1898232003dd448791f6d6b222731e5a, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/b7b06ed62cbe8eca9103c3d111564e6c/B/91193976fe4c4e15b19485e66f2f6950, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/b7b06ed62cbe8eca9103c3d111564e6c/B/8044099aed62448096cf6b59026c51f9] into tmpdir=hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/b7b06ed62cbe8eca9103c3d111564e6c/.tmp, totalSize=34.8 K 2023-04-15 02:54:05,224 DEBUG [RS:0;jenkins-hbase20:34681-shortCompactions-0] compactions.Compactor(207): Compacting 8f538e98aa65491ebfab8d01af624c27, keycount=150, bloomtype=ROW, size=25.1 K, encoding=NONE, compression=NONE, seqNum=50, earliestPutTs=1681527243474 2023-04-15 02:54:05,224 DEBUG [RS:0;jenkins-hbase20:34681-longCompactions-0] compactions.Compactor(207): Compacting 1898232003dd448791f6d6b222731e5a, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=16, earliestPutTs=1681527243418 2023-04-15 02:54:05,224 DEBUG [RS:0;jenkins-hbase20:34681-shortCompactions-0] compactions.Compactor(207): Compacting 7219c23fbaee4d32b1f84582e92030c0, keycount=200, bloomtype=ROW, size=31.9 K, encoding=NONE, compression=NONE, seqNum=62, earliestPutTs=1681527244084 2023-04-15 02:54:05,224 DEBUG [RS:0;jenkins-hbase20:34681-longCompactions-0] compactions.Compactor(207): Compacting 91193976fe4c4e15b19485e66f2f6950, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=50, earliestPutTs=1681527243474 2023-04-15 02:54:05,242 DEBUG [RS:0;jenkins-hbase20:34681-longCompactions-0] compactions.Compactor(207): Compacting 8044099aed62448096cf6b59026c51f9, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=62, earliestPutTs=1681527244084 2023-04-15 02:54:05,247 INFO [RS:0;jenkins-hbase20:34681-longCompactions-0] throttle.PressureAwareThroughputController(145): b7b06ed62cbe8eca9103c3d111564e6c#B#compaction#111 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-15 02:54:05,261 DEBUG [Thread-864] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-15 02:54:05,261 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-15 02:54:05,263 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-15 02:54:05,266 INFO [RS:0;jenkins-hbase20:34681-shortCompactions-0] throttle.PressureAwareThroughputController(145): b7b06ed62cbe8eca9103c3d111564e6c#A#compaction#112 average throughput is 0.99 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-15 02:54:05,266 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-15 02:54:05,267 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-15 02:54:05,267 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-15 02:54:05,267 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-15 02:54:05,267 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-15 02:54:05,267 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-15 02:54:05,267 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:05,267 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:54:05,269 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-15 02:54:05,269 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:05,269 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-15 02:54:05,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-15 02:54:05,269 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:05,269 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-15 02:54:05,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:54:05,269 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:05,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-15 02:54:05,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:54:05,270 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-15 02:54:05,270 DEBUG [RS:0;jenkins-hbase20:34681-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/b7b06ed62cbe8eca9103c3d111564e6c/.tmp/B/9de6a2bcd583484f8cb8a66f0b62cdbe as hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/b7b06ed62cbe8eca9103c3d111564e6c/B/9de6a2bcd583484f8cb8a66f0b62cdbe 2023-04-15 02:54:05,270 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-15 02:54:05,270 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-15 02:54:05,270 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-15 02:54:05,270 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-15 02:54:05,271 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-15 02:54:05,271 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-15 02:54:05,271 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681527240182.b7b06ed62cbe8eca9103c3d111564e6c. 2023-04-15 02:54:05,271 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681527240182.b7b06ed62cbe8eca9103c3d111564e6c. started... 2023-04-15 02:54:05,271 INFO [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing b7b06ed62cbe8eca9103c3d111564e6c 3/3 column families, dataSize=118.53 KB heapSize=335.38 KB 2023-04-15 02:54:05,271 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK b7b06ed62cbe8eca9103c3d111564e6c, store=A 2023-04-15 02:54:05,271 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-15 02:54:05,272 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK b7b06ed62cbe8eca9103c3d111564e6c, store=B 2023-04-15 02:54:05,272 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-15 02:54:05,272 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK b7b06ed62cbe8eca9103c3d111564e6c, store=C 2023-04-15 02:54:05,272 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-15 02:54:05,278 INFO [RS:0;jenkins-hbase20:34681-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in b7b06ed62cbe8eca9103c3d111564e6c/B of b7b06ed62cbe8eca9103c3d111564e6c into 9de6a2bcd583484f8cb8a66f0b62cdbe(size=11.7 K), total size for store is 11.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-15 02:54:05,278 DEBUG [RS:0;jenkins-hbase20:34681-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for b7b06ed62cbe8eca9103c3d111564e6c: 2023-04-15 02:54:05,278 INFO [RS:0;jenkins-hbase20:34681-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681527240182.b7b06ed62cbe8eca9103c3d111564e6c., storeName=b7b06ed62cbe8eca9103c3d111564e6c/B, priority=13, startTime=1681527245221; duration=0sec 2023-04-15 02:54:05,278 DEBUG [RS:0;jenkins-hbase20:34681-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-15 02:54:05,278 DEBUG [RS:0;jenkins-hbase20:34681-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-15 02:54:05,280 DEBUG [RS:0;jenkins-hbase20:34681-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 35658 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-15 02:54:05,280 DEBUG [RS:0;jenkins-hbase20:34681-longCompactions-0] regionserver.HStore(1912): b7b06ed62cbe8eca9103c3d111564e6c/C is initiating minor compaction (all files) 2023-04-15 02:54:05,280 INFO [RS:0;jenkins-hbase20:34681-longCompactions-0] regionserver.HRegion(2259): Starting compaction of b7b06ed62cbe8eca9103c3d111564e6c/C in TestAcidGuarantees,,1681527240182.b7b06ed62cbe8eca9103c3d111564e6c. 2023-04-15 02:54:05,280 INFO [RS:0;jenkins-hbase20:34681-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/b7b06ed62cbe8eca9103c3d111564e6c/C/353790da0a4f4819b1625c8eb352666e, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/b7b06ed62cbe8eca9103c3d111564e6c/C/deb3773513d94fcf990a9633a3cfe8f2, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/b7b06ed62cbe8eca9103c3d111564e6c/C/38eaea4cbf6e46cbbe3107d8924ce952] into tmpdir=hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/b7b06ed62cbe8eca9103c3d111564e6c/.tmp, totalSize=34.8 K 2023-04-15 02:54:05,281 DEBUG [RS:0;jenkins-hbase20:34681-longCompactions-0] compactions.Compactor(207): Compacting 353790da0a4f4819b1625c8eb352666e, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=16, earliestPutTs=1681527243418 2023-04-15 02:54:05,281 DEBUG [RS:0;jenkins-hbase20:34681-longCompactions-0] compactions.Compactor(207): Compacting deb3773513d94fcf990a9633a3cfe8f2, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=50, earliestPutTs=1681527243474 2023-04-15 02:54:05,282 DEBUG [RS:0;jenkins-hbase20:34681-longCompactions-0] compactions.Compactor(207): Compacting 38eaea4cbf6e46cbbe3107d8924ce952, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=62, earliestPutTs=1681527244084 2023-04-15 02:54:05,303 INFO [RS:0;jenkins-hbase20:34681-longCompactions-0] throttle.PressureAwareThroughputController(145): b7b06ed62cbe8eca9103c3d111564e6c#C#compaction#114 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-15 02:54:05,310 INFO [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230415065542fc348c48a28f3acb41347d7764 to hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230415065542fc348c48a28f3acb41347d7764 2023-04-15 02:54:05,380 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34681] regionserver.HRegion(9158): Flush requested on b7b06ed62cbe8eca9103c3d111564e6c 2023-04-15 02:54:05,380 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681527240182.b7b06ed62cbe8eca9103c3d111564e6c. as already flushing 2023-04-15 02:54:05,402 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b7b06ed62cbe8eca9103c3d111564e6c, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:54:05,402 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34681] ipc.CallRunner(144): callId: 49 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:39132 deadline: 1681527305398, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b7b06ed62cbe8eca9103c3d111564e6c, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:05,404 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b7b06ed62cbe8eca9103c3d111564e6c, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:54:05,404 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34681] ipc.CallRunner(144): callId: 54 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:39106 deadline: 1681527305402, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b7b06ed62cbe8eca9103c3d111564e6c, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:05,506 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b7b06ed62cbe8eca9103c3d111564e6c, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:54:05,506 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34681] ipc.CallRunner(144): callId: 51 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:39132 deadline: 1681527305503, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b7b06ed62cbe8eca9103c3d111564e6c, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:05,506 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b7b06ed62cbe8eca9103c3d111564e6c, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:54:05,506 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34681] ipc.CallRunner(144): callId: 56 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:39106 deadline: 1681527305505, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b7b06ed62cbe8eca9103c3d111564e6c, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:05,606 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b7b06ed62cbe8eca9103c3d111564e6c, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:54:05,606 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34681] ipc.CallRunner(144): callId: 21 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:39116 deadline: 1681527305603, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b7b06ed62cbe8eca9103c3d111564e6c, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:05,608 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b7b06ed62cbe8eca9103c3d111564e6c, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:54:05,608 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34681] ipc.CallRunner(144): callId: 22 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:39108 deadline: 1681527305606, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b7b06ed62cbe8eca9103c3d111564e6c, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:05,608 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b7b06ed62cbe8eca9103c3d111564e6c, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:54:05,608 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34681] ipc.CallRunner(144): callId: 19 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:39142 deadline: 1681527305608, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b7b06ed62cbe8eca9103c3d111564e6c, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:05,707 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b7b06ed62cbe8eca9103c3d111564e6c, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:54:05,707 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34681] ipc.CallRunner(144): callId: 58 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:39106 deadline: 1681527305707, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b7b06ed62cbe8eca9103c3d111564e6c, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:05,709 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b7b06ed62cbe8eca9103c3d111564e6c, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:54:05,710 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34681] ipc.CallRunner(144): callId: 53 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:39132 deadline: 1681527305708, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b7b06ed62cbe8eca9103c3d111564e6c, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:05,716 DEBUG [RS:0;jenkins-hbase20:34681-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/b7b06ed62cbe8eca9103c3d111564e6c/.tmp/C/737e570c76d24414ac42ba9b0bd82ec0 as hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/b7b06ed62cbe8eca9103c3d111564e6c/C/737e570c76d24414ac42ba9b0bd82ec0 2023-04-15 02:54:05,721 INFO [RS:0;jenkins-hbase20:34681-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in b7b06ed62cbe8eca9103c3d111564e6c/C of b7b06ed62cbe8eca9103c3d111564e6c into 737e570c76d24414ac42ba9b0bd82ec0(size=11.7 K), total size for store is 11.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-15 02:54:05,721 DEBUG [RS:0;jenkins-hbase20:34681-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for b7b06ed62cbe8eca9103c3d111564e6c: 2023-04-15 02:54:05,721 INFO [RS:0;jenkins-hbase20:34681-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681527240182.b7b06ed62cbe8eca9103c3d111564e6c., storeName=b7b06ed62cbe8eca9103c3d111564e6c/C, priority=13, startTime=1681527245222; duration=0sec 2023-04-15 02:54:05,721 DEBUG [RS:0;jenkins-hbase20:34681-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-15 02:54:05,728 INFO [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=96, memsize=40.3 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/b7b06ed62cbe8eca9103c3d111564e6c/.tmp/A/add8b3c6156c490a94ee447b44124c71 2023-04-15 02:54:05,744 INFO [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=40.25 KB at sequenceid=96 (bloomFilter=true), to=hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/b7b06ed62cbe8eca9103c3d111564e6c/.tmp/B/21086d2253f94dcea031b131bdd490f8 2023-04-15 02:54:06,008 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b7b06ed62cbe8eca9103c3d111564e6c, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:54:06,008 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34681] ipc.CallRunner(144): callId: 60 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:39106 deadline: 1681527306008, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b7b06ed62cbe8eca9103c3d111564e6c, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:06,013 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b7b06ed62cbe8eca9103c3d111564e6c, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:54:06,013 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34681] ipc.CallRunner(144): callId: 55 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:39132 deadline: 1681527306011, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b7b06ed62cbe8eca9103c3d111564e6c, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:06,133 DEBUG [RS:0;jenkins-hbase20:34681-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/b7b06ed62cbe8eca9103c3d111564e6c/.tmp/A/891f9e563f9b42f4aa0368a0f6313a84 as hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/b7b06ed62cbe8eca9103c3d111564e6c/A/891f9e563f9b42f4aa0368a0f6313a84 2023-04-15 02:54:06,140 INFO [RS:0;jenkins-hbase20:34681-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in b7b06ed62cbe8eca9103c3d111564e6c/A of b7b06ed62cbe8eca9103c3d111564e6c into 891f9e563f9b42f4aa0368a0f6313a84(size=25.2 K), total size for store is 25.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-15 02:54:06,140 DEBUG [RS:0;jenkins-hbase20:34681-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for b7b06ed62cbe8eca9103c3d111564e6c: 2023-04-15 02:54:06,140 INFO [RS:0;jenkins-hbase20:34681-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681527240182.b7b06ed62cbe8eca9103c3d111564e6c., storeName=b7b06ed62cbe8eca9103c3d111564e6c/A, priority=13, startTime=1681527245221; duration=0sec 2023-04-15 02:54:06,140 DEBUG [RS:0;jenkins-hbase20:34681-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-15 02:54:06,155 INFO [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=96 (bloomFilter=true), to=hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/b7b06ed62cbe8eca9103c3d111564e6c/.tmp/C/d664f390a19847bf93af9bc550f95fda 2023-04-15 02:54:06,160 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/b7b06ed62cbe8eca9103c3d111564e6c/.tmp/A/add8b3c6156c490a94ee447b44124c71 as hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/b7b06ed62cbe8eca9103c3d111564e6c/A/add8b3c6156c490a94ee447b44124c71 2023-04-15 02:54:06,165 INFO [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/b7b06ed62cbe8eca9103c3d111564e6c/A/add8b3c6156c490a94ee447b44124c71, entries=150, sequenceid=96, filesize=25.1 K 2023-04-15 02:54:06,165 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/b7b06ed62cbe8eca9103c3d111564e6c/.tmp/B/21086d2253f94dcea031b131bdd490f8 as hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/b7b06ed62cbe8eca9103c3d111564e6c/B/21086d2253f94dcea031b131bdd490f8 2023-04-15 02:54:06,170 INFO [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/b7b06ed62cbe8eca9103c3d111564e6c/B/21086d2253f94dcea031b131bdd490f8, entries=150, sequenceid=96, filesize=11.6 K 2023-04-15 02:54:06,171 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/b7b06ed62cbe8eca9103c3d111564e6c/.tmp/C/d664f390a19847bf93af9bc550f95fda as hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/b7b06ed62cbe8eca9103c3d111564e6c/C/d664f390a19847bf93af9bc550f95fda 2023-04-15 02:54:06,180 INFO [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/b7b06ed62cbe8eca9103c3d111564e6c/C/d664f390a19847bf93af9bc550f95fda, entries=150, sequenceid=96, filesize=11.6 K 2023-04-15 02:54:06,181 INFO [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~118.53 KB/121370, heapSize ~335.33 KB/343376, currentSize=60.38 KB/61830 for b7b06ed62cbe8eca9103c3d111564e6c in 910ms, sequenceid=96, compaction requested=false 2023-04-15 02:54:06,181 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for b7b06ed62cbe8eca9103c3d111564e6c: 2023-04-15 02:54:06,181 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681527240182.b7b06ed62cbe8eca9103c3d111564e6c. 2023-04-15 02:54:06,181 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-15 02:54:06,181 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-15 02:54:06,181 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:06,181 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-15 02:54:06,181 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34681,1681527209133' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-15 02:54:06,182 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:06,182 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:06,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:06,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:54:06,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:54:06,182 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:06,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:54:06,182 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-15 02:54:06,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:54:06,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:54:06,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:06,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:54:06,184 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34681,1681527209133' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-15 02:54:06,184 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6f83c031[Count = 0] remaining members to acquire global barrier 2023-04-15 02:54:06,184 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-15 02:54:06,184 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:06,185 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:06,185 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:06,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:06,185 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-15 02:54:06,185 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-15 02:54:06,185 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34681,1681527209133' in zk 2023-04-15 02:54:06,186 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:06,186 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-15 02:54:06,186 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:06,186 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-15 02:54:06,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:06,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:54:06,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:54:06,186 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-15 02:54:06,191 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-15 02:54:06,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:54:06,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:54:06,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:54:06,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:06,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:54:06,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:54:06,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:06,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34681,1681527209133': 2023-04-15 02:54:06,200 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34681,1681527209133' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-15 02:54:06,200 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-15 02:54:06,200 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-15 02:54:06,200 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-15 02:54:06,200 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-15 02:54:06,200 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-15 02:54:06,201 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:06,201 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:06,201 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-15 02:54:06,201 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:06,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:06,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:54:06,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:54:06,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:06,202 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:06,202 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-15 02:54:06,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-15 02:54:06,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:54:06,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:06,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:54:06,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:54:06,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:54:06,203 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:06,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:06,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:54:06,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:54:06,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:06,206 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:06,206 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:54:06,206 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-15 02:54:06,206 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:54:06,206 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-15 02:54:06,206 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-15 02:54:06,206 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-15 02:54:06,206 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:06,207 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-15 02:54:06,207 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:06,207 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:06,207 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-15 02:54:06,207 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:06,207 DEBUG [Thread-864] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-15 02:54:06,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-15 02:54:06,207 DEBUG [Thread-864] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-15 02:54:06,207 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-15 02:54:06,207 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-15 02:54:06,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-15 02:54:06,307 DEBUG [Thread-864] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-15 02:54:06,308 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-15 02:54:06,310 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-15 02:54:06,311 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-15 02:54:06,311 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-15 02:54:06,311 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-15 02:54:06,311 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-15 02:54:06,311 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-15 02:54:06,311 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-15 02:54:06,312 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:06,312 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:54:06,312 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:06,312 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-15 02:54:06,313 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-15 02:54:06,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-15 02:54:06,313 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:06,313 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-15 02:54:06,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:54:06,313 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:06,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-15 02:54:06,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:54:06,314 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-15 02:54:06,314 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-15 02:54:06,314 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-15 02:54:06,314 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-15 02:54:06,314 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-15 02:54:06,314 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-15 02:54:06,314 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-15 02:54:06,314 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-15 02:54:06,314 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681527240182.b7b06ed62cbe8eca9103c3d111564e6c. 2023-04-15 02:54:06,314 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681527240182.b7b06ed62cbe8eca9103c3d111564e6c. started... 2023-04-15 02:54:06,315 INFO [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing b7b06ed62cbe8eca9103c3d111564e6c 3/3 column families, dataSize=60.38 KB heapSize=180.05 KB 2023-04-15 02:54:06,315 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK b7b06ed62cbe8eca9103c3d111564e6c, store=A 2023-04-15 02:54:06,315 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-15 02:54:06,315 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK b7b06ed62cbe8eca9103c3d111564e6c, store=B 2023-04-15 02:54:06,315 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-15 02:54:06,315 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK b7b06ed62cbe8eca9103c3d111564e6c, store=C 2023-04-15 02:54:06,315 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-15 02:54:06,517 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34681] regionserver.HRegion(9158): Flush requested on b7b06ed62cbe8eca9103c3d111564e6c 2023-04-15 02:54:06,517 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681527240182.b7b06ed62cbe8eca9103c3d111564e6c. as already flushing 2023-04-15 02:54:06,577 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b7b06ed62cbe8eca9103c3d111564e6c, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:54:06,579 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34681] ipc.CallRunner(144): callId: 65 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:39132 deadline: 1681527306576, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b7b06ed62cbe8eca9103c3d111564e6c, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:06,582 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b7b06ed62cbe8eca9103c3d111564e6c, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:54:06,582 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34681] ipc.CallRunner(144): callId: 71 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:39106 deadline: 1681527306577, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b7b06ed62cbe8eca9103c3d111564e6c, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:06,682 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b7b06ed62cbe8eca9103c3d111564e6c, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:54:06,682 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34681] ipc.CallRunner(144): callId: 67 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:39132 deadline: 1681527306680, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b7b06ed62cbe8eca9103c3d111564e6c, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:06,684 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b7b06ed62cbe8eca9103c3d111564e6c, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:54:06,684 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34681] ipc.CallRunner(144): callId: 73 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:39106 deadline: 1681527306683, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b7b06ed62cbe8eca9103c3d111564e6c, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:06,736 INFO [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304156304ea2649aa4d86b4ab10dbd3a25d7a to hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304156304ea2649aa4d86b4ab10dbd3a25d7a 2023-04-15 02:54:06,887 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b7b06ed62cbe8eca9103c3d111564e6c, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:54:06,887 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34681] ipc.CallRunner(144): callId: 69 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:39132 deadline: 1681527306884, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b7b06ed62cbe8eca9103c3d111564e6c, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:06,887 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b7b06ed62cbe8eca9103c3d111564e6c, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:54:06,887 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34681] ipc.CallRunner(144): callId: 75 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:39106 deadline: 1681527306885, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b7b06ed62cbe8eca9103c3d111564e6c, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:07,141 INFO [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=111, memsize=20.1 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/b7b06ed62cbe8eca9103c3d111564e6c/.tmp/A/edf1440b49904cc88b8ed1f7e7cb63d4 2023-04-15 02:54:07,191 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b7b06ed62cbe8eca9103c3d111564e6c, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:54:07,191 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34681] ipc.CallRunner(144): callId: 77 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:39106 deadline: 1681527307189, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b7b06ed62cbe8eca9103c3d111564e6c, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:07,191 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b7b06ed62cbe8eca9103c3d111564e6c, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:54:07,191 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34681] ipc.CallRunner(144): callId: 71 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:39132 deadline: 1681527307190, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b7b06ed62cbe8eca9103c3d111564e6c, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:07,555 INFO [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=111 (bloomFilter=true), to=hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/b7b06ed62cbe8eca9103c3d111564e6c/.tmp/B/cfd9ac1c518c4f72ae895b29915a6ea1 2023-04-15 02:54:07,610 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b7b06ed62cbe8eca9103c3d111564e6c, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:54:07,610 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34681] ipc.CallRunner(144): callId: 21 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:39142 deadline: 1681527307609, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b7b06ed62cbe8eca9103c3d111564e6c, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:07,611 DEBUG [Thread-856] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4128 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b7b06ed62cbe8eca9103c3d111564e6c, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_1' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1681527240182.b7b06ed62cbe8eca9103c3d111564e6c., hostname=jenkins-hbase20.apache.org,34681,1681527209133, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b7b06ed62cbe8eca9103c3d111564e6c, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor31.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b7b06ed62cbe8eca9103c3d111564e6c, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-15 02:54:07,627 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b7b06ed62cbe8eca9103c3d111564e6c, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:54:07,628 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34681] ipc.CallRunner(144): callId: 23 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:39116 deadline: 1681527307626, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b7b06ed62cbe8eca9103c3d111564e6c, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:07,628 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b7b06ed62cbe8eca9103c3d111564e6c, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:54:07,628 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34681] ipc.CallRunner(144): callId: 24 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:39108 deadline: 1681527307628, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b7b06ed62cbe8eca9103c3d111564e6c, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:07,630 DEBUG [Thread-854] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4150 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b7b06ed62cbe8eca9103c3d111564e6c, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_0' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1681527240182.b7b06ed62cbe8eca9103c3d111564e6c., hostname=jenkins-hbase20.apache.org,34681,1681527209133, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b7b06ed62cbe8eca9103c3d111564e6c, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor31.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b7b06ed62cbe8eca9103c3d111564e6c, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-15 02:54:07,631 DEBUG [Thread-858] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4154 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b7b06ed62cbe8eca9103c3d111564e6c, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_0' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1681527240182.b7b06ed62cbe8eca9103c3d111564e6c., hostname=jenkins-hbase20.apache.org,34681,1681527209133, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b7b06ed62cbe8eca9103c3d111564e6c, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor31.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b7b06ed62cbe8eca9103c3d111564e6c, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-15 02:54:07,696 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b7b06ed62cbe8eca9103c3d111564e6c, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:54:07,696 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34681] ipc.CallRunner(144): callId: 73 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:39132 deadline: 1681527307694, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b7b06ed62cbe8eca9103c3d111564e6c, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:07,696 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b7b06ed62cbe8eca9103c3d111564e6c, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:54:07,696 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34681] ipc.CallRunner(144): callId: 79 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:39106 deadline: 1681527307695, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b7b06ed62cbe8eca9103c3d111564e6c, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:07,979 INFO [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=111 (bloomFilter=true), to=hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/b7b06ed62cbe8eca9103c3d111564e6c/.tmp/C/8b5d68def4f844bcbbbb22de6f29644a 2023-04-15 02:54:07,984 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/b7b06ed62cbe8eca9103c3d111564e6c/.tmp/A/edf1440b49904cc88b8ed1f7e7cb63d4 as hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/b7b06ed62cbe8eca9103c3d111564e6c/A/edf1440b49904cc88b8ed1f7e7cb63d4 2023-04-15 02:54:07,988 INFO [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/b7b06ed62cbe8eca9103c3d111564e6c/A/edf1440b49904cc88b8ed1f7e7cb63d4, entries=100, sequenceid=111, filesize=18.4 K 2023-04-15 02:54:07,989 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/b7b06ed62cbe8eca9103c3d111564e6c/.tmp/B/cfd9ac1c518c4f72ae895b29915a6ea1 as hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/b7b06ed62cbe8eca9103c3d111564e6c/B/cfd9ac1c518c4f72ae895b29915a6ea1 2023-04-15 02:54:07,992 INFO [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/b7b06ed62cbe8eca9103c3d111564e6c/B/cfd9ac1c518c4f72ae895b29915a6ea1, entries=100, sequenceid=111, filesize=9.3 K 2023-04-15 02:54:07,993 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/b7b06ed62cbe8eca9103c3d111564e6c/.tmp/C/8b5d68def4f844bcbbbb22de6f29644a as hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/b7b06ed62cbe8eca9103c3d111564e6c/C/8b5d68def4f844bcbbbb22de6f29644a 2023-04-15 02:54:07,998 INFO [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/b7b06ed62cbe8eca9103c3d111564e6c/C/8b5d68def4f844bcbbbb22de6f29644a, entries=100, sequenceid=111, filesize=9.3 K 2023-04-15 02:54:07,999 INFO [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~60.38 KB/61830, heapSize ~180 KB/184320, currentSize=114.05 KB/116790 for b7b06ed62cbe8eca9103c3d111564e6c in 1684ms, sequenceid=111, compaction requested=true 2023-04-15 02:54:07,999 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for b7b06ed62cbe8eca9103c3d111564e6c: 2023-04-15 02:54:07,999 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681527240182.b7b06ed62cbe8eca9103c3d111564e6c. 2023-04-15 02:54:07,999 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-15 02:54:07,999 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-15 02:54:07,999 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:07,999 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-15 02:54:07,999 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34681,1681527209133' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-15 02:54:08,000 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:08,000 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:08,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:08,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:54:08,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:54:08,000 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:08,000 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-15 02:54:08,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:54:08,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:54:08,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:54:08,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:08,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:54:08,001 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34681,1681527209133' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-15 02:54:08,001 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-15 02:54:08,001 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@db3cd0[Count = 0] remaining members to acquire global barrier 2023-04-15 02:54:08,001 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:08,002 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:08,002 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:08,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:08,002 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-15 02:54:08,002 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-15 02:54:08,002 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34681,1681527209133' in zk 2023-04-15 02:54:08,002 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:08,002 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-15 02:54:08,003 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-15 02:54:08,003 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:08,003 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-15 02:54:08,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:08,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:54:08,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:54:08,003 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-15 02:54:08,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:54:08,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:54:08,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:54:08,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:08,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:54:08,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:54:08,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:08,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34681,1681527209133': 2023-04-15 02:54:08,005 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34681,1681527209133' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-15 02:54:08,005 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-15 02:54:08,005 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-15 02:54:08,005 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-15 02:54:08,005 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-15 02:54:08,005 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-15 02:54:08,006 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:08,006 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:08,006 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:08,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:08,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:54:08,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:54:08,006 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-15 02:54:08,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:08,006 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-15 02:54:08,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-15 02:54:08,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:54:08,006 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:08,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:08,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:54:08,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:54:08,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:54:08,007 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:08,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:08,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:54:08,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:54:08,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:08,009 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:08,009 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:54:08,009 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:54:08,009 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-15 02:54:08,009 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-15 02:54:08,009 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-15 02:54:08,009 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:08,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-15 02:54:08,009 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-15 02:54:08,009 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-15 02:54:08,009 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-15 02:54:08,009 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-15 02:54:08,009 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:08,009 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:08,009 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-15 02:54:08,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-15 02:54:08,009 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:08,009 DEBUG [Thread-864] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-15 02:54:08,009 DEBUG [Thread-864] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-15 02:54:08,109 DEBUG [Thread-864] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-15 02:54:08,110 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-15 02:54:08,111 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-15 02:54:08,112 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-15 02:54:08,113 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-15 02:54:08,113 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-15 02:54:08,113 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-15 02:54:08,113 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-15 02:54:08,113 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-15 02:54:08,113 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:08,113 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:54:08,114 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-15 02:54:08,114 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:08,114 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-15 02:54:08,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-15 02:54:08,114 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:08,114 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-15 02:54:08,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:54:08,114 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:08,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-15 02:54:08,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:54:08,115 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-15 02:54:08,115 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-15 02:54:08,115 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-15 02:54:08,115 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-15 02:54:08,115 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-15 02:54:08,115 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-15 02:54:08,115 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-15 02:54:08,115 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-15 02:54:08,115 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681527240182.b7b06ed62cbe8eca9103c3d111564e6c. 2023-04-15 02:54:08,115 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681527240182.b7b06ed62cbe8eca9103c3d111564e6c. started... 2023-04-15 02:54:08,115 INFO [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing b7b06ed62cbe8eca9103c3d111564e6c 3/3 column families, dataSize=114.05 KB heapSize=339.42 KB 2023-04-15 02:54:08,116 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK b7b06ed62cbe8eca9103c3d111564e6c, store=A 2023-04-15 02:54:08,116 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-15 02:54:08,116 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK b7b06ed62cbe8eca9103c3d111564e6c, store=B 2023-04-15 02:54:08,116 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-15 02:54:08,116 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK b7b06ed62cbe8eca9103c3d111564e6c, store=C 2023-04-15 02:54:08,116 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-15 02:54:08,530 INFO [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230415eb6dfbfce2e946da9ce20208b2efeab4 to hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230415eb6dfbfce2e946da9ce20208b2efeab4 2023-04-15 02:54:08,699 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34681] regionserver.HRegion(9158): Flush requested on b7b06ed62cbe8eca9103c3d111564e6c 2023-04-15 02:54:08,699 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681527240182.b7b06ed62cbe8eca9103c3d111564e6c. as already flushing 2023-04-15 02:54:08,713 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b7b06ed62cbe8eca9103c3d111564e6c, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:54:08,714 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34681] ipc.CallRunner(144): callId: 80 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:39132 deadline: 1681527308710, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b7b06ed62cbe8eca9103c3d111564e6c, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:08,714 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b7b06ed62cbe8eca9103c3d111564e6c, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:54:08,714 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34681] ipc.CallRunner(144): callId: 85 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:39106 deadline: 1681527308711, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b7b06ed62cbe8eca9103c3d111564e6c, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:08,763 WARN [HBase-Metrics2-1] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties 2023-04-15 02:54:08,815 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b7b06ed62cbe8eca9103c3d111564e6c, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:54:08,815 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34681] ipc.CallRunner(144): callId: 82 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:39132 deadline: 1681527308814, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b7b06ed62cbe8eca9103c3d111564e6c, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:08,817 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b7b06ed62cbe8eca9103c3d111564e6c, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:54:08,817 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34681] ipc.CallRunner(144): callId: 87 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:39106 deadline: 1681527308815, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b7b06ed62cbe8eca9103c3d111564e6c, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:08,964 INFO [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=131, memsize=38.0 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/b7b06ed62cbe8eca9103c3d111564e6c/.tmp/A/5df9409e91d84edc91bc186464378c47 2023-04-15 02:54:09,019 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b7b06ed62cbe8eca9103c3d111564e6c, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:54:09,019 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34681] ipc.CallRunner(144): callId: 84 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:39132 deadline: 1681527309017, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b7b06ed62cbe8eca9103c3d111564e6c, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:09,019 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b7b06ed62cbe8eca9103c3d111564e6c, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:54:09,019 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34681] ipc.CallRunner(144): callId: 89 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:39106 deadline: 1681527309018, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b7b06ed62cbe8eca9103c3d111564e6c, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:09,321 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b7b06ed62cbe8eca9103c3d111564e6c, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:54:09,321 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34681] ipc.CallRunner(144): callId: 86 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:39132 deadline: 1681527309321, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b7b06ed62cbe8eca9103c3d111564e6c, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:09,323 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b7b06ed62cbe8eca9103c3d111564e6c, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:54:09,323 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34681] ipc.CallRunner(144): callId: 91 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:39106 deadline: 1681527309322, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b7b06ed62cbe8eca9103c3d111564e6c, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:09,381 INFO [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=131 (bloomFilter=true), to=hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/b7b06ed62cbe8eca9103c3d111564e6c/.tmp/B/474c82d609a447279075788f36feb259 2023-04-15 02:54:09,805 INFO [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=131 (bloomFilter=true), to=hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/b7b06ed62cbe8eca9103c3d111564e6c/.tmp/C/dccb7202a6a94ef58345640fce8cc84f 2023-04-15 02:54:09,809 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/b7b06ed62cbe8eca9103c3d111564e6c/.tmp/A/5df9409e91d84edc91bc186464378c47 as hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/b7b06ed62cbe8eca9103c3d111564e6c/A/5df9409e91d84edc91bc186464378c47 2023-04-15 02:54:09,813 INFO [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/b7b06ed62cbe8eca9103c3d111564e6c/A/5df9409e91d84edc91bc186464378c47, entries=150, sequenceid=131, filesize=25.2 K 2023-04-15 02:54:09,814 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/b7b06ed62cbe8eca9103c3d111564e6c/.tmp/B/474c82d609a447279075788f36feb259 as hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/b7b06ed62cbe8eca9103c3d111564e6c/B/474c82d609a447279075788f36feb259 2023-04-15 02:54:09,818 INFO [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/b7b06ed62cbe8eca9103c3d111564e6c/B/474c82d609a447279075788f36feb259, entries=150, sequenceid=131, filesize=11.7 K 2023-04-15 02:54:09,819 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/b7b06ed62cbe8eca9103c3d111564e6c/.tmp/C/dccb7202a6a94ef58345640fce8cc84f as hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/b7b06ed62cbe8eca9103c3d111564e6c/C/dccb7202a6a94ef58345640fce8cc84f 2023-04-15 02:54:09,823 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b7b06ed62cbe8eca9103c3d111564e6c, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:54:09,823 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34681] ipc.CallRunner(144): callId: 88 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:39132 deadline: 1681527309822, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b7b06ed62cbe8eca9103c3d111564e6c, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:09,825 INFO [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/b7b06ed62cbe8eca9103c3d111564e6c/C/dccb7202a6a94ef58345640fce8cc84f, entries=150, sequenceid=131, filesize=11.7 K 2023-04-15 02:54:09,826 INFO [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~114.05 KB/116790, heapSize ~339.38 KB/347520, currentSize=60.38 KB/61830 for b7b06ed62cbe8eca9103c3d111564e6c in 1710ms, sequenceid=131, compaction requested=true 2023-04-15 02:54:09,826 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for b7b06ed62cbe8eca9103c3d111564e6c: 2023-04-15 02:54:09,826 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681527240182.b7b06ed62cbe8eca9103c3d111564e6c. 2023-04-15 02:54:09,826 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-15 02:54:09,826 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-15 02:54:09,826 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:09,826 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-15 02:54:09,826 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34681,1681527209133' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-15 02:54:09,827 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:09,827 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:09,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:09,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:54:09,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:54:09,827 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:09,827 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-15 02:54:09,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:54:09,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:54:09,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:54:09,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:09,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:54:09,828 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34681,1681527209133' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-15 02:54:09,828 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@34c392c8[Count = 0] remaining members to acquire global barrier 2023-04-15 02:54:09,828 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-15 02:54:09,828 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:09,829 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:09,829 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:09,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:09,829 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-15 02:54:09,829 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-15 02:54:09,829 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34681,1681527209133' in zk 2023-04-15 02:54:09,829 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:09,829 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-15 02:54:09,830 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-15 02:54:09,830 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:09,830 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-15 02:54:09,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:09,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:54:09,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:54:09,830 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-15 02:54:09,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:54:09,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:54:09,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:54:09,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:09,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:54:09,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:54:09,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:09,831 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34681] regionserver.HRegion(9158): Flush requested on b7b06ed62cbe8eca9103c3d111564e6c 2023-04-15 02:54:09,831 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing b7b06ed62cbe8eca9103c3d111564e6c 3/3 column families, dataSize=67.09 KB heapSize=199.97 KB 2023-04-15 02:54:09,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34681,1681527209133': 2023-04-15 02:54:09,832 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34681,1681527209133' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-15 02:54:09,832 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-15 02:54:09,832 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-15 02:54:09,832 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-15 02:54:09,832 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK b7b06ed62cbe8eca9103c3d111564e6c, store=A 2023-04-15 02:54:09,832 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-15 02:54:09,832 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-15 02:54:09,832 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-15 02:54:09,832 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK b7b06ed62cbe8eca9103c3d111564e6c, store=B 2023-04-15 02:54:09,832 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-15 02:54:09,832 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK b7b06ed62cbe8eca9103c3d111564e6c, store=C 2023-04-15 02:54:09,832 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-15 02:54:09,835 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:09,835 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:09,835 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-15 02:54:09,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:09,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:54:09,835 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:09,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:54:09,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:09,835 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-15 02:54:09,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-15 02:54:09,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:09,836 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:09,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:54:09,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:54:09,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:54:09,837 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:09,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:54:09,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:09,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:54:09,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:54:09,848 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:09,848 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:54:09,848 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-15 02:54:09,848 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-15 02:54:09,848 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:54:09,848 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-15 02:54:09,848 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-15 02:54:09,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-15 02:54:09,848 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:09,848 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:09,849 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:09,849 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:09,848 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-15 02:54:09,848 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-15 02:54:09,848 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-15 02:54:09,849 DEBUG [Thread-864] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-15 02:54:09,849 DEBUG [Thread-864] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-15 02:54:09,849 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-15 02:54:09,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-15 02:54:09,862 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230415441230a350ff4baa9f843583b71686f1 to hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230415441230a350ff4baa9f843583b71686f1 2023-04-15 02:54:09,887 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b7b06ed62cbe8eca9103c3d111564e6c, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:54:09,888 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34681] ipc.CallRunner(144): callId: 110 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:39106 deadline: 1681527309885, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b7b06ed62cbe8eca9103c3d111564e6c, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:09,949 DEBUG [Thread-864] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-15 02:54:09,949 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-15 02:54:09,951 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-15 02:54:09,952 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-15 02:54:09,952 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-15 02:54:09,952 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-15 02:54:09,952 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-15 02:54:09,952 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-15 02:54:09,952 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-15 02:54:09,953 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:09,953 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:54:09,953 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-15 02:54:09,953 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:09,953 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-15 02:54:09,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-15 02:54:09,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:54:09,954 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:09,954 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-15 02:54:09,954 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:09,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-15 02:54:09,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:54:09,956 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-15 02:54:09,956 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-15 02:54:09,956 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-15 02:54:09,956 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-15 02:54:09,956 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-15 02:54:09,956 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-15 02:54:09,957 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-15 02:54:09,957 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-15 02:54:09,957 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681527240182.b7b06ed62cbe8eca9103c3d111564e6c. 2023-04-15 02:54:09,957 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681527240182.b7b06ed62cbe8eca9103c3d111564e6c. started... 2023-04-15 02:54:09,957 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681527240182.b7b06ed62cbe8eca9103c3d111564e6c. as already flushing 2023-04-15 02:54:09,957 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681527240182.b7b06ed62cbe8eca9103c3d111564e6c. 2023-04-15 02:54:09,957 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-15 02:54:09,957 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-15 02:54:09,957 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:09,957 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-15 02:54:09,957 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34681,1681527209133' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-15 02:54:09,958 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:09,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:09,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:54:09,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:54:09,959 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:09,959 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:09,959 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-15 02:54:09,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:54:09,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:54:09,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:54:09,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:09,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:54:09,964 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34681,1681527209133' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-15 02:54:09,964 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@25177c61[Count = 0] remaining members to acquire global barrier 2023-04-15 02:54:09,964 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-15 02:54:09,964 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:09,965 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:09,965 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:09,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:09,965 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-15 02:54:09,965 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-15 02:54:09,965 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34681,1681527209133' in zk 2023-04-15 02:54:09,965 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:09,965 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-15 02:54:09,966 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:09,966 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-15 02:54:09,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:09,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:54:09,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:54:09,966 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-15 02:54:09,966 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-15 02:54:09,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:54:09,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:54:09,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:54:09,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:09,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:54:09,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:54:09,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:09,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34681,1681527209133': 2023-04-15 02:54:09,969 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34681,1681527209133' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-15 02:54:09,969 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-15 02:54:09,970 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-15 02:54:09,970 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-15 02:54:09,970 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-15 02:54:09,970 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-15 02:54:09,977 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:09,977 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:09,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:09,977 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:09,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:09,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:54:09,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:54:09,977 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-15 02:54:09,977 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-15 02:54:09,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-15 02:54:09,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:09,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:54:09,979 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:09,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:54:09,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:54:09,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:54:09,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:09,980 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:09,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:54:09,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:54:09,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:09,986 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-15 02:54:09,986 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:09,986 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:54:09,986 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:54:09,986 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-15 02:54:09,986 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-15 02:54:09,986 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-15 02:54:09,986 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:09,986 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:09,986 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:09,986 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:09,986 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-15 02:54:09,986 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-15 02:54:09,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-15 02:54:09,986 DEBUG [Thread-864] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-15 02:54:09,987 DEBUG [Thread-864] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-15 02:54:09,987 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-15 02:54:09,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-15 02:54:09,992 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b7b06ed62cbe8eca9103c3d111564e6c, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:54:09,992 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34681] ipc.CallRunner(144): callId: 112 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:39106 deadline: 1681527309989, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b7b06ed62cbe8eca9103c3d111564e6c, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:10,087 DEBUG [Thread-864] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-15 02:54:10,087 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-15 02:54:10,089 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-15 02:54:10,090 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-15 02:54:10,090 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-15 02:54:10,090 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-15 02:54:10,090 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-15 02:54:10,090 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-15 02:54:10,091 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-15 02:54:10,091 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:10,091 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:54:10,092 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-15 02:54:10,092 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:10,092 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-15 02:54:10,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-15 02:54:10,092 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:10,092 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-15 02:54:10,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:54:10,092 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:10,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-15 02:54:10,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:54:10,092 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-15 02:54:10,093 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-15 02:54:10,093 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-15 02:54:10,093 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-15 02:54:10,093 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-15 02:54:10,093 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-15 02:54:10,093 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-15 02:54:10,093 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-15 02:54:10,093 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681527240182.b7b06ed62cbe8eca9103c3d111564e6c. 2023-04-15 02:54:10,093 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681527240182.b7b06ed62cbe8eca9103c3d111564e6c. started... 2023-04-15 02:54:10,093 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681527240182.b7b06ed62cbe8eca9103c3d111564e6c. as already flushing 2023-04-15 02:54:10,093 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681527240182.b7b06ed62cbe8eca9103c3d111564e6c. 2023-04-15 02:54:10,093 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-15 02:54:10,093 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-15 02:54:10,093 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:10,093 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-15 02:54:10,093 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34681,1681527209133' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-15 02:54:10,094 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:10,094 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:10,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:10,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:54:10,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:54:10,094 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:10,094 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-15 02:54:10,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:54:10,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:54:10,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:54:10,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:10,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:54:10,096 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34681,1681527209133' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-15 02:54:10,096 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5464a2ba[Count = 0] remaining members to acquire global barrier 2023-04-15 02:54:10,096 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-15 02:54:10,096 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:10,096 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:10,096 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:10,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:10,096 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-15 02:54:10,096 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-15 02:54:10,096 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34681,1681527209133' in zk 2023-04-15 02:54:10,096 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:10,096 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-15 02:54:10,097 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:10,097 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-15 02:54:10,097 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-15 02:54:10,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:10,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:54:10,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:54:10,097 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-15 02:54:10,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:54:10,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:54:10,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:54:10,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:10,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:54:10,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:54:10,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:10,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34681,1681527209133': 2023-04-15 02:54:10,100 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34681,1681527209133' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-15 02:54:10,100 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-15 02:54:10,100 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-15 02:54:10,100 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-15 02:54:10,100 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-15 02:54:10,100 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-15 02:54:10,101 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:10,101 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:10,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:10,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:54:10,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:54:10,101 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:10,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:10,101 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-15 02:54:10,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:54:10,102 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:10,102 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-15 02:54:10,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-15 02:54:10,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:54:10,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:10,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:54:10,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:54:10,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:10,103 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:10,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:54:10,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:54:10,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:10,104 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-15 02:54:10,104 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:10,104 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-15 02:54:10,104 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-15 02:54:10,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-15 02:54:10,104 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:54:10,104 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-15 02:54:10,105 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-15 02:54:10,104 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-15 02:54:10,105 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-15 02:54:10,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-15 02:54:10,105 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:54:10,105 DEBUG [Thread-864] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-15 02:54:10,105 DEBUG [Thread-864] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-15 02:54:10,105 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:10,105 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:10,105 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:10,105 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:10,195 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b7b06ed62cbe8eca9103c3d111564e6c, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:54:10,195 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34681] ipc.CallRunner(144): callId: 114 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:39106 deadline: 1681527310193, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b7b06ed62cbe8eca9103c3d111564e6c, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:10,205 DEBUG [Thread-864] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-15 02:54:10,206 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-15 02:54:10,207 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-15 02:54:10,208 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-15 02:54:10,208 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-15 02:54:10,208 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-15 02:54:10,208 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-15 02:54:10,208 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-15 02:54:10,208 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-15 02:54:10,208 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:10,208 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:54:10,209 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:10,209 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-15 02:54:10,209 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-15 02:54:10,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-15 02:54:10,209 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:10,209 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-15 02:54:10,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:54:10,210 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:10,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-15 02:54:10,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:54:10,210 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-15 02:54:10,210 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-15 02:54:10,210 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-15 02:54:10,210 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-15 02:54:10,210 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-15 02:54:10,210 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-15 02:54:10,210 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-15 02:54:10,210 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-15 02:54:10,210 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681527240182.b7b06ed62cbe8eca9103c3d111564e6c. 2023-04-15 02:54:10,210 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681527240182.b7b06ed62cbe8eca9103c3d111564e6c. started... 2023-04-15 02:54:10,210 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681527240182.b7b06ed62cbe8eca9103c3d111564e6c. as already flushing 2023-04-15 02:54:10,210 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681527240182.b7b06ed62cbe8eca9103c3d111564e6c. 2023-04-15 02:54:10,210 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-15 02:54:10,210 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-15 02:54:10,211 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:10,211 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-15 02:54:10,211 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34681,1681527209133' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-15 02:54:10,211 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:10,211 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:10,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:10,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:54:10,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:54:10,212 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:10,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:54:10,212 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-15 02:54:10,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:54:10,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:54:10,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:10,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:54:10,212 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34681,1681527209133' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-15 02:54:10,213 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@d00295a[Count = 0] remaining members to acquire global barrier 2023-04-15 02:54:10,213 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-15 02:54:10,213 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:10,213 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:10,213 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:10,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:10,213 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-15 02:54:10,213 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-15 02:54:10,213 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34681,1681527209133' in zk 2023-04-15 02:54:10,213 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:10,213 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-15 02:54:10,214 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:10,214 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-15 02:54:10,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:10,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:54:10,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:54:10,214 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-15 02:54:10,214 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-15 02:54:10,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:54:10,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:54:10,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:54:10,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:10,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:54:10,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:54:10,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:10,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34681,1681527209133': 2023-04-15 02:54:10,216 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34681,1681527209133' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-15 02:54:10,216 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-15 02:54:10,216 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-15 02:54:10,216 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-15 02:54:10,216 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-15 02:54:10,216 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-15 02:54:10,216 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:10,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:10,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:54:10,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:54:10,216 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:10,217 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-15 02:54:10,217 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:10,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:10,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:54:10,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:54:10,217 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-15 02:54:10,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-15 02:54:10,217 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:10,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:10,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:54:10,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:54:10,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:10,218 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:10,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:54:10,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:54:10,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:10,220 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:10,220 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:54:10,220 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:54:10,220 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:10,220 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:10,220 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:10,220 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:10,220 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-15 02:54:10,220 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-15 02:54:10,220 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-15 02:54:10,220 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-15 02:54:10,220 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-15 02:54:10,220 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-15 02:54:10,220 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-15 02:54:10,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-15 02:54:10,220 DEBUG [Thread-864] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-15 02:54:10,221 DEBUG [Thread-864] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-15 02:54:10,223 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-15 02:54:10,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-15 02:54:10,287 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=144, memsize=22.4 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/b7b06ed62cbe8eca9103c3d111564e6c/.tmp/A/153faaeb5918489c8f014d09aeb0238e 2023-04-15 02:54:10,321 DEBUG [Thread-864] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-15 02:54:10,321 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-15 02:54:10,322 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-15 02:54:10,323 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-15 02:54:10,323 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-15 02:54:10,323 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-15 02:54:10,323 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-15 02:54:10,323 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-15 02:54:10,323 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-15 02:54:10,324 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:10,324 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:54:10,324 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-15 02:54:10,324 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:10,324 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-15 02:54:10,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-15 02:54:10,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:54:10,326 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:10,326 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-15 02:54:10,326 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:10,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-15 02:54:10,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:54:10,326 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-15 02:54:10,327 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-15 02:54:10,327 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-15 02:54:10,327 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-15 02:54:10,327 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-15 02:54:10,327 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-15 02:54:10,329 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-15 02:54:10,329 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-15 02:54:10,329 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681527240182.b7b06ed62cbe8eca9103c3d111564e6c. 2023-04-15 02:54:10,329 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681527240182.b7b06ed62cbe8eca9103c3d111564e6c. started... 2023-04-15 02:54:10,329 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681527240182.b7b06ed62cbe8eca9103c3d111564e6c. as already flushing 2023-04-15 02:54:10,329 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681527240182.b7b06ed62cbe8eca9103c3d111564e6c. 2023-04-15 02:54:10,329 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-15 02:54:10,329 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-15 02:54:10,329 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:10,329 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-15 02:54:10,329 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34681,1681527209133' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-15 02:54:10,337 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:10,337 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:10,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:10,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:54:10,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:54:10,337 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:10,338 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-15 02:54:10,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:54:10,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:54:10,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:54:10,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:10,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:54:10,339 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34681,1681527209133' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-15 02:54:10,339 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@257dafe0[Count = 0] remaining members to acquire global barrier 2023-04-15 02:54:10,339 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-15 02:54:10,339 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:10,341 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:10,342 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:10,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:10,342 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-15 02:54:10,342 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-15 02:54:10,342 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:10,342 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-15 02:54:10,342 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34681,1681527209133' in zk 2023-04-15 02:54:10,343 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-15 02:54:10,343 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:10,343 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-15 02:54:10,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:10,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:54:10,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:54:10,343 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-15 02:54:10,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:54:10,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:54:10,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:54:10,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:10,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:54:10,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:54:10,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:10,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34681,1681527209133': 2023-04-15 02:54:10,344 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34681,1681527209133' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-15 02:54:10,344 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-15 02:54:10,344 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-15 02:54:10,344 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-15 02:54:10,344 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-15 02:54:10,344 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-15 02:54:10,345 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:10,345 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:10,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:10,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:54:10,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:54:10,345 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-15 02:54:10,345 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:10,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:10,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:54:10,350 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-15 02:54:10,350 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:10,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-15 02:54:10,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:54:10,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:10,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:54:10,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:54:10,351 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:10,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:10,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:54:10,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:54:10,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:10,358 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:10,358 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:54:10,358 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:54:10,358 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-15 02:54:10,358 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-15 02:54:10,358 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-15 02:54:10,358 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-15 02:54:10,359 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-15 02:54:10,358 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:10,359 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:10,359 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:10,359 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:10,358 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-15 02:54:10,358 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-15 02:54:10,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-15 02:54:10,359 DEBUG [Thread-864] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-15 02:54:10,359 DEBUG [Thread-864] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-15 02:54:10,359 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-15 02:54:10,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-15 02:54:10,459 DEBUG [Thread-864] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-15 02:54:10,460 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-15 02:54:10,461 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-15 02:54:10,465 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-15 02:54:10,465 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-15 02:54:10,465 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-15 02:54:10,465 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-15 02:54:10,465 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-15 02:54:10,465 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-15 02:54:10,465 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:10,466 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:54:10,467 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:10,467 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-15 02:54:10,467 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-15 02:54:10,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-15 02:54:10,467 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:10,467 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-15 02:54:10,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:54:10,467 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:10,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-15 02:54:10,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:54:10,468 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-15 02:54:10,468 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-15 02:54:10,468 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-15 02:54:10,468 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-15 02:54:10,468 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-15 02:54:10,468 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-15 02:54:10,468 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-15 02:54:10,468 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-15 02:54:10,468 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681527240182.b7b06ed62cbe8eca9103c3d111564e6c. 2023-04-15 02:54:10,468 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681527240182.b7b06ed62cbe8eca9103c3d111564e6c. started... 2023-04-15 02:54:10,468 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681527240182.b7b06ed62cbe8eca9103c3d111564e6c. as already flushing 2023-04-15 02:54:10,468 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681527240182.b7b06ed62cbe8eca9103c3d111564e6c. 2023-04-15 02:54:10,468 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-15 02:54:10,468 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-15 02:54:10,468 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:10,468 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-15 02:54:10,468 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34681,1681527209133' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-15 02:54:10,469 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:10,469 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:10,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:10,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:54:10,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:54:10,469 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:10,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:54:10,469 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-15 02:54:10,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:54:10,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:54:10,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:10,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:54:10,470 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34681,1681527209133' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-15 02:54:10,470 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-15 02:54:10,470 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3b972b5a[Count = 0] remaining members to acquire global barrier 2023-04-15 02:54:10,470 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:10,471 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:10,471 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:10,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:10,471 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-15 02:54:10,471 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-15 02:54:10,471 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:10,471 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-15 02:54:10,471 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34681,1681527209133' in zk 2023-04-15 02:54:10,472 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:10,472 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-15 02:54:10,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:10,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:54:10,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:54:10,472 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-15 02:54:10,472 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-15 02:54:10,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:54:10,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:54:10,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:54:10,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:10,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:54:10,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:54:10,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:10,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34681,1681527209133': 2023-04-15 02:54:10,474 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34681,1681527209133' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-15 02:54:10,474 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-15 02:54:10,474 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-15 02:54:10,474 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-15 02:54:10,474 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-15 02:54:10,474 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-15 02:54:10,475 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:10,475 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:10,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:10,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:54:10,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:54:10,475 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:10,475 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-15 02:54:10,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:10,475 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:10,475 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-15 02:54:10,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-15 02:54:10,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:54:10,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:10,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:54:10,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:54:10,476 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:10,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:54:10,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:10,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:54:10,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:54:10,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:10,477 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-15 02:54:10,477 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:10,477 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-15 02:54:10,477 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-15 02:54:10,477 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-15 02:54:10,477 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-15 02:54:10,477 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:54:10,477 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-15 02:54:10,477 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:54:10,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-15 02:54:10,477 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:10,478 DEBUG [Thread-864] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-15 02:54:10,478 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:10,478 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-15 02:54:10,478 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:10,478 DEBUG [Thread-864] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-15 02:54:10,478 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:10,478 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-15 02:54:10,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-15 02:54:10,498 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b7b06ed62cbe8eca9103c3d111564e6c, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:54:10,498 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34681] ipc.CallRunner(144): callId: 116 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:39106 deadline: 1681527310498, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b7b06ed62cbe8eca9103c3d111564e6c, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:10,578 DEBUG [Thread-864] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-15 02:54:10,578 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-15 02:54:10,579 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-15 02:54:10,580 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-15 02:54:10,580 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-15 02:54:10,580 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-15 02:54:10,580 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-15 02:54:10,580 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-15 02:54:10,581 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-15 02:54:10,581 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:10,581 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:54:10,581 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:10,581 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-15 02:54:10,582 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-15 02:54:10,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-15 02:54:10,582 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:10,582 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-15 02:54:10,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:54:10,582 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:10,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-15 02:54:10,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:54:10,582 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-15 02:54:10,582 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-15 02:54:10,583 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-15 02:54:10,583 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-15 02:54:10,583 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-15 02:54:10,583 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-15 02:54:10,583 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-15 02:54:10,583 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-15 02:54:10,583 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681527240182.b7b06ed62cbe8eca9103c3d111564e6c. 2023-04-15 02:54:10,583 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681527240182.b7b06ed62cbe8eca9103c3d111564e6c. started... 2023-04-15 02:54:10,583 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681527240182.b7b06ed62cbe8eca9103c3d111564e6c. as already flushing 2023-04-15 02:54:10,583 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681527240182.b7b06ed62cbe8eca9103c3d111564e6c. 2023-04-15 02:54:10,583 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-15 02:54:10,583 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-15 02:54:10,583 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:10,583 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-15 02:54:10,583 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34681,1681527209133' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-15 02:54:10,584 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:10,584 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:10,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:10,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:54:10,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:54:10,584 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:10,584 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-15 02:54:10,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:54:10,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:54:10,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:54:10,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:10,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:54:10,586 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34681,1681527209133' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-15 02:54:10,586 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4a972fe0[Count = 0] remaining members to acquire global barrier 2023-04-15 02:54:10,586 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-15 02:54:10,586 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:10,587 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:10,587 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:10,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:10,587 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-15 02:54:10,587 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:10,587 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-15 02:54:10,587 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-15 02:54:10,587 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34681,1681527209133' in zk 2023-04-15 02:54:10,588 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-15 02:54:10,588 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:10,588 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-15 02:54:10,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:10,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:54:10,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:54:10,588 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-15 02:54:10,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:54:10,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:54:10,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:54:10,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:10,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:54:10,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:54:10,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:10,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34681,1681527209133': 2023-04-15 02:54:10,590 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34681,1681527209133' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-15 02:54:10,590 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-15 02:54:10,590 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-15 02:54:10,590 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-15 02:54:10,590 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-15 02:54:10,590 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-15 02:54:10,591 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:10,591 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-15 02:54:10,591 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:10,591 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:10,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:10,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:10,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:54:10,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:54:10,592 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:10,592 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-15 02:54:10,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:54:10,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-15 02:54:10,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:54:10,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:10,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:54:10,593 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:10,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:54:10,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:10,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:54:10,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:54:10,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:10,594 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:10,594 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:54:10,594 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:54:10,594 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-15 02:54:10,594 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-15 02:54:10,594 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:10,594 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-15 02:54:10,594 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-15 02:54:10,594 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-15 02:54:10,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-15 02:54:10,594 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-15 02:54:10,594 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:10,594 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:10,594 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:10,594 DEBUG [Thread-864] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-15 02:54:10,595 DEBUG [Thread-864] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-15 02:54:10,595 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-15 02:54:10,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-15 02:54:10,595 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-15 02:54:10,695 DEBUG [Thread-864] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-15 02:54:10,695 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-15 02:54:10,696 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-15 02:54:10,697 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-15 02:54:10,697 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-15 02:54:10,697 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-15 02:54:10,698 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-15 02:54:10,698 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-15 02:54:10,698 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-15 02:54:10,698 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:10,698 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:54:10,699 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=144 (bloomFilter=true), to=hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/b7b06ed62cbe8eca9103c3d111564e6c/.tmp/B/f562373783fb4d908240d252b4e6387a 2023-04-15 02:54:10,703 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-15 02:54:10,703 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-15 02:54:10,703 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:10,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-15 02:54:10,703 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:10,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:54:10,703 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-15 02:54:10,704 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:10,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-15 02:54:10,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:54:10,704 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-15 02:54:10,704 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-15 02:54:10,704 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-15 02:54:10,704 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-15 02:54:10,704 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-15 02:54:10,705 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-15 02:54:10,705 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-15 02:54:10,705 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-15 02:54:10,705 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681527240182.b7b06ed62cbe8eca9103c3d111564e6c. 2023-04-15 02:54:10,705 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681527240182.b7b06ed62cbe8eca9103c3d111564e6c. started... 2023-04-15 02:54:10,705 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681527240182.b7b06ed62cbe8eca9103c3d111564e6c. as already flushing 2023-04-15 02:54:10,705 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681527240182.b7b06ed62cbe8eca9103c3d111564e6c. 2023-04-15 02:54:10,705 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-15 02:54:10,705 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-15 02:54:10,705 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:10,705 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-15 02:54:10,705 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34681,1681527209133' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-15 02:54:10,706 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:10,706 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:10,706 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-15 02:54:10,706 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:10,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:10,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:54:10,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:54:10,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:54:10,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:54:10,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:54:10,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:10,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:54:10,708 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34681,1681527209133' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-15 02:54:10,708 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@46d19511[Count = 0] remaining members to acquire global barrier 2023-04-15 02:54:10,708 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-15 02:54:10,708 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:10,709 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:10,709 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:10,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:10,709 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-15 02:54:10,709 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-15 02:54:10,709 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:10,709 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-15 02:54:10,709 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34681,1681527209133' in zk 2023-04-15 02:54:10,710 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=144 (bloomFilter=true), to=hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/b7b06ed62cbe8eca9103c3d111564e6c/.tmp/C/058660d60eb74ece847d2441e5be3182 2023-04-15 02:54:10,710 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-15 02:54:10,710 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:10,710 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-15 02:54:10,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:10,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:54:10,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:54:10,710 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-15 02:54:10,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:54:10,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:54:10,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:54:10,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:10,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:54:10,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:54:10,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:10,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34681,1681527209133': 2023-04-15 02:54:10,712 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34681,1681527209133' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-15 02:54:10,712 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-15 02:54:10,712 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-15 02:54:10,712 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-15 02:54:10,712 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-15 02:54:10,712 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-15 02:54:10,713 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:10,713 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:10,713 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-15 02:54:10,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:10,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:54:10,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:54:10,713 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:10,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:10,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:54:10,714 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-15 02:54:10,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-15 02:54:10,714 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:10,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:54:10,714 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/b7b06ed62cbe8eca9103c3d111564e6c/.tmp/A/153faaeb5918489c8f014d09aeb0238e as hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/b7b06ed62cbe8eca9103c3d111564e6c/A/153faaeb5918489c8f014d09aeb0238e 2023-04-15 02:54:10,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:10,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:54:10,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:54:10,715 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:10,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:10,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:54:10,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:54:10,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:10,717 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:10,717 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:54:10,717 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:54:10,717 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:10,717 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-15 02:54:10,717 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-15 02:54:10,717 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-15 02:54:10,717 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-15 02:54:10,717 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-15 02:54:10,718 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-15 02:54:10,717 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:10,718 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-15 02:54:10,718 DEBUG [Thread-864] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-15 02:54:10,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-15 02:54:10,718 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:10,718 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:10,718 DEBUG [Thread-864] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-15 02:54:10,718 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-15 02:54:10,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-15 02:54:10,718 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/b7b06ed62cbe8eca9103c3d111564e6c/A/153faaeb5918489c8f014d09aeb0238e, entries=200, sequenceid=144, filesize=32.1 K 2023-04-15 02:54:10,719 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/b7b06ed62cbe8eca9103c3d111564e6c/.tmp/B/f562373783fb4d908240d252b4e6387a as hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/b7b06ed62cbe8eca9103c3d111564e6c/B/f562373783fb4d908240d252b4e6387a 2023-04-15 02:54:10,726 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/b7b06ed62cbe8eca9103c3d111564e6c/B/f562373783fb4d908240d252b4e6387a, entries=150, sequenceid=144, filesize=11.8 K 2023-04-15 02:54:10,727 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/b7b06ed62cbe8eca9103c3d111564e6c/.tmp/C/058660d60eb74ece847d2441e5be3182 as hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/b7b06ed62cbe8eca9103c3d111564e6c/C/058660d60eb74ece847d2441e5be3182 2023-04-15 02:54:10,731 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/b7b06ed62cbe8eca9103c3d111564e6c/C/058660d60eb74ece847d2441e5be3182, entries=150, sequenceid=144, filesize=11.8 K 2023-04-15 02:54:10,732 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~67.09 KB/68700, heapSize ~199.92 KB/204720, currentSize=107.34 KB/109920 for b7b06ed62cbe8eca9103c3d111564e6c in 901ms, sequenceid=144, compaction requested=true 2023-04-15 02:54:10,732 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for b7b06ed62cbe8eca9103c3d111564e6c: 2023-04-15 02:54:10,732 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-15 02:54:10,732 DEBUG [RS:0;jenkins-hbase20:34681-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-04-15 02:54:10,733 DEBUG [RS:0;jenkins-hbase20:34681-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-04-15 02:54:10,733 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-15 02:54:10,733 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-15 02:54:10,735 DEBUG [RS:0;jenkins-hbase20:34681-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 129125 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-04-15 02:54:10,736 DEBUG [RS:0;jenkins-hbase20:34681-shortCompactions-0] regionserver.HStore(1912): b7b06ed62cbe8eca9103c3d111564e6c/A is initiating minor compaction (all files) 2023-04-15 02:54:10,736 INFO [RS:0;jenkins-hbase20:34681-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of b7b06ed62cbe8eca9103c3d111564e6c/A in TestAcidGuarantees,,1681527240182.b7b06ed62cbe8eca9103c3d111564e6c. 2023-04-15 02:54:10,736 INFO [RS:0;jenkins-hbase20:34681-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/b7b06ed62cbe8eca9103c3d111564e6c/A/891f9e563f9b42f4aa0368a0f6313a84, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/b7b06ed62cbe8eca9103c3d111564e6c/A/add8b3c6156c490a94ee447b44124c71, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/b7b06ed62cbe8eca9103c3d111564e6c/A/edf1440b49904cc88b8ed1f7e7cb63d4, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/b7b06ed62cbe8eca9103c3d111564e6c/A/5df9409e91d84edc91bc186464378c47, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/b7b06ed62cbe8eca9103c3d111564e6c/A/153faaeb5918489c8f014d09aeb0238e] into tmpdir=hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/b7b06ed62cbe8eca9103c3d111564e6c/.tmp, totalSize=126.1 K 2023-04-15 02:54:10,736 DEBUG [RS:0;jenkins-hbase20:34681-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 57439 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-04-15 02:54:10,736 DEBUG [RS:0;jenkins-hbase20:34681-longCompactions-0] regionserver.HStore(1912): b7b06ed62cbe8eca9103c3d111564e6c/B is initiating minor compaction (all files) 2023-04-15 02:54:10,736 DEBUG [RS:0;jenkins-hbase20:34681-shortCompactions-0] compactions.Compactor(207): Compacting 891f9e563f9b42f4aa0368a0f6313a84, keycount=150, bloomtype=ROW, size=25.2 K, encoding=NONE, compression=NONE, seqNum=62, earliestPutTs=1681527244084 2023-04-15 02:54:10,736 INFO [RS:0;jenkins-hbase20:34681-longCompactions-0] regionserver.HRegion(2259): Starting compaction of b7b06ed62cbe8eca9103c3d111564e6c/B in TestAcidGuarantees,,1681527240182.b7b06ed62cbe8eca9103c3d111564e6c. 2023-04-15 02:54:10,736 INFO [RS:0;jenkins-hbase20:34681-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/b7b06ed62cbe8eca9103c3d111564e6c/B/9de6a2bcd583484f8cb8a66f0b62cdbe, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/b7b06ed62cbe8eca9103c3d111564e6c/B/21086d2253f94dcea031b131bdd490f8, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/b7b06ed62cbe8eca9103c3d111564e6c/B/cfd9ac1c518c4f72ae895b29915a6ea1, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/b7b06ed62cbe8eca9103c3d111564e6c/B/474c82d609a447279075788f36feb259, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/b7b06ed62cbe8eca9103c3d111564e6c/B/f562373783fb4d908240d252b4e6387a] into tmpdir=hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/b7b06ed62cbe8eca9103c3d111564e6c/.tmp, totalSize=56.1 K 2023-04-15 02:54:10,737 DEBUG [RS:0;jenkins-hbase20:34681-shortCompactions-0] compactions.Compactor(207): Compacting add8b3c6156c490a94ee447b44124c71, keycount=150, bloomtype=ROW, size=25.1 K, encoding=NONE, compression=NONE, seqNum=96, earliestPutTs=1681527244766 2023-04-15 02:54:10,737 DEBUG [RS:0;jenkins-hbase20:34681-longCompactions-0] compactions.Compactor(207): Compacting 9de6a2bcd583484f8cb8a66f0b62cdbe, keycount=150, bloomtype=ROW, size=11.7 K, encoding=NONE, compression=NONE, seqNum=62, earliestPutTs=1681527244084 2023-04-15 02:54:10,737 DEBUG [RS:0;jenkins-hbase20:34681-shortCompactions-0] compactions.Compactor(207): Compacting edf1440b49904cc88b8ed1f7e7cb63d4, keycount=100, bloomtype=ROW, size=18.4 K, encoding=NONE, compression=NONE, seqNum=111, earliestPutTs=1681527245396 2023-04-15 02:54:10,737 DEBUG [RS:0;jenkins-hbase20:34681-longCompactions-0] compactions.Compactor(207): Compacting 21086d2253f94dcea031b131bdd490f8, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=96, earliestPutTs=1681527244766 2023-04-15 02:54:10,738 DEBUG [RS:0;jenkins-hbase20:34681-shortCompactions-0] compactions.Compactor(207): Compacting 5df9409e91d84edc91bc186464378c47, keycount=150, bloomtype=ROW, size=25.2 K, encoding=NONE, compression=NONE, seqNum=131, earliestPutTs=1681527246571 2023-04-15 02:54:10,738 DEBUG [RS:0;jenkins-hbase20:34681-longCompactions-0] compactions.Compactor(207): Compacting cfd9ac1c518c4f72ae895b29915a6ea1, keycount=100, bloomtype=ROW, size=9.3 K, encoding=NONE, compression=NONE, seqNum=111, earliestPutTs=1681527245396 2023-04-15 02:54:10,738 DEBUG [RS:0;jenkins-hbase20:34681-shortCompactions-0] compactions.Compactor(207): Compacting 153faaeb5918489c8f014d09aeb0238e, keycount=200, bloomtype=ROW, size=32.1 K, encoding=NONE, compression=NONE, seqNum=144, earliestPutTs=1681527248703 2023-04-15 02:54:10,738 DEBUG [RS:0;jenkins-hbase20:34681-longCompactions-0] compactions.Compactor(207): Compacting 474c82d609a447279075788f36feb259, keycount=150, bloomtype=ROW, size=11.7 K, encoding=NONE, compression=NONE, seqNum=131, earliestPutTs=1681527246571 2023-04-15 02:54:10,739 DEBUG [RS:0;jenkins-hbase20:34681-longCompactions-0] compactions.Compactor(207): Compacting f562373783fb4d908240d252b4e6387a, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=144, earliestPutTs=1681527248703 2023-04-15 02:54:10,748 INFO [RS:0;jenkins-hbase20:34681-longCompactions-0] throttle.PressureAwareThroughputController(145): b7b06ed62cbe8eca9103c3d111564e6c#B#compaction#126 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-15 02:54:10,754 INFO [RS:0;jenkins-hbase20:34681-shortCompactions-0] throttle.PressureAwareThroughputController(145): b7b06ed62cbe8eca9103c3d111564e6c#A#compaction#127 average throughput is 2.46 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-15 02:54:10,818 DEBUG [Thread-864] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-15 02:54:10,819 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-15 02:54:10,820 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-15 02:54:10,821 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-15 02:54:10,821 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-15 02:54:10,821 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-15 02:54:10,821 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-15 02:54:10,821 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-15 02:54:10,821 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-15 02:54:10,821 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:10,821 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:54:10,822 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:10,822 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-15 02:54:10,822 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-15 02:54:10,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-15 02:54:10,822 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:10,822 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-15 02:54:10,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:54:10,822 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:10,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-15 02:54:10,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:54:10,823 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-15 02:54:10,823 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-15 02:54:10,823 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-15 02:54:10,823 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-15 02:54:10,823 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-15 02:54:10,823 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-15 02:54:10,823 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-15 02:54:10,823 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-15 02:54:10,823 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681527240182.b7b06ed62cbe8eca9103c3d111564e6c. 2023-04-15 02:54:10,823 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681527240182.b7b06ed62cbe8eca9103c3d111564e6c. started... 2023-04-15 02:54:10,823 INFO [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing b7b06ed62cbe8eca9103c3d111564e6c 3/3 column families, dataSize=107.34 KB heapSize=319.50 KB 2023-04-15 02:54:10,823 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK b7b06ed62cbe8eca9103c3d111564e6c, store=A 2023-04-15 02:54:10,823 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-15 02:54:10,823 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK b7b06ed62cbe8eca9103c3d111564e6c, store=B 2023-04-15 02:54:10,823 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-15 02:54:10,824 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK b7b06ed62cbe8eca9103c3d111564e6c, store=C 2023-04-15 02:54:10,824 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-15 02:54:10,829 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34681] regionserver.HRegion(9158): Flush requested on b7b06ed62cbe8eca9103c3d111564e6c 2023-04-15 02:54:10,829 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681527240182.b7b06ed62cbe8eca9103c3d111564e6c. as already flushing 2023-04-15 02:54:10,839 INFO [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230415c37111470da7478abf1a1fdf2f9829a3 to hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230415c37111470da7478abf1a1fdf2f9829a3 2023-04-15 02:54:10,845 INFO [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=163, memsize=35.8 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/b7b06ed62cbe8eca9103c3d111564e6c/.tmp/A/4f734e0147ea4a368d6ec7845146f73b 2023-04-15 02:54:10,859 INFO [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=163 (bloomFilter=true), to=hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/b7b06ed62cbe8eca9103c3d111564e6c/.tmp/B/11454a78c616407d849c8a14e7d1674a 2023-04-15 02:54:10,863 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b7b06ed62cbe8eca9103c3d111564e6c, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:54:10,863 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34681] ipc.CallRunner(144): callId: 100 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:39132 deadline: 1681527310859, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b7b06ed62cbe8eca9103c3d111564e6c, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:10,870 INFO [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=163 (bloomFilter=true), to=hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/b7b06ed62cbe8eca9103c3d111564e6c/.tmp/C/be56c7c9f6194b33b06978f39ebd5f62 2023-04-15 02:54:10,874 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/b7b06ed62cbe8eca9103c3d111564e6c/.tmp/A/4f734e0147ea4a368d6ec7845146f73b as hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/b7b06ed62cbe8eca9103c3d111564e6c/A/4f734e0147ea4a368d6ec7845146f73b 2023-04-15 02:54:10,880 INFO [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/b7b06ed62cbe8eca9103c3d111564e6c/A/4f734e0147ea4a368d6ec7845146f73b, entries=150, sequenceid=163, filesize=25.3 K 2023-04-15 02:54:10,881 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/b7b06ed62cbe8eca9103c3d111564e6c/.tmp/B/11454a78c616407d849c8a14e7d1674a as hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/b7b06ed62cbe8eca9103c3d111564e6c/B/11454a78c616407d849c8a14e7d1674a 2023-04-15 02:54:10,885 INFO [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/b7b06ed62cbe8eca9103c3d111564e6c/B/11454a78c616407d849c8a14e7d1674a, entries=150, sequenceid=163, filesize=11.8 K 2023-04-15 02:54:10,885 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/b7b06ed62cbe8eca9103c3d111564e6c/.tmp/C/be56c7c9f6194b33b06978f39ebd5f62 as hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/b7b06ed62cbe8eca9103c3d111564e6c/C/be56c7c9f6194b33b06978f39ebd5f62 2023-04-15 02:54:10,891 INFO [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/b7b06ed62cbe8eca9103c3d111564e6c/C/be56c7c9f6194b33b06978f39ebd5f62, entries=150, sequenceid=163, filesize=11.8 K 2023-04-15 02:54:10,891 INFO [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~107.34 KB/109920, heapSize ~319.45 KB/327120, currentSize=67.09 KB/68700 for b7b06ed62cbe8eca9103c3d111564e6c in 68ms, sequenceid=163, compaction requested=true 2023-04-15 02:54:10,891 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for b7b06ed62cbe8eca9103c3d111564e6c: 2023-04-15 02:54:10,892 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681527240182.b7b06ed62cbe8eca9103c3d111564e6c. 2023-04-15 02:54:10,892 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-15 02:54:10,892 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-15 02:54:10,892 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:10,894 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-15 02:54:10,894 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34681,1681527209133' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-15 02:54:10,896 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:10,896 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:10,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:10,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:54:10,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:54:10,896 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:10,896 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-15 02:54:10,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:54:10,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:54:10,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:54:10,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:10,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:54:10,897 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34681,1681527209133' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-15 02:54:10,897 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-15 02:54:10,897 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@e1b3084[Count = 0] remaining members to acquire global barrier 2023-04-15 02:54:10,897 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:10,898 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:10,898 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:10,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:10,898 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-15 02:54:10,898 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-15 02:54:10,898 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34681,1681527209133' in zk 2023-04-15 02:54:10,898 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:10,898 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-15 02:54:10,899 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:10,899 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-15 02:54:10,899 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-15 02:54:10,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:10,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:54:10,899 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-15 02:54:10,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:54:10,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:54:10,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:54:10,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:54:10,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:10,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:54:10,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:54:10,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:10,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34681,1681527209133': 2023-04-15 02:54:10,901 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34681,1681527209133' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-15 02:54:10,901 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-15 02:54:10,901 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-15 02:54:10,901 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-15 02:54:10,901 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-15 02:54:10,901 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-15 02:54:10,902 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:10,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:10,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:54:10,902 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:10,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:54:10,902 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:10,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:10,902 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-15 02:54:10,903 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-15 02:54:10,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-15 02:54:10,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:54:10,904 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:10,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:10,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:54:10,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:54:10,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:54:10,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:10,905 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:10,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:54:10,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:54:10,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:10,907 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:10,907 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:54:10,907 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:54:10,907 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:10,907 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-15 02:54:10,907 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-15 02:54:10,907 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-15 02:54:10,907 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-15 02:54:10,907 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:10,907 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-15 02:54:10,908 DEBUG [Thread-864] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-15 02:54:10,908 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:10,908 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:10,907 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-15 02:54:10,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-15 02:54:10,908 DEBUG [Thread-864] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-15 02:54:10,908 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-15 02:54:10,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-15 02:54:10,966 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34681] regionserver.HRegion(9158): Flush requested on b7b06ed62cbe8eca9103c3d111564e6c 2023-04-15 02:54:10,966 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing b7b06ed62cbe8eca9103c3d111564e6c 3/3 column families, dataSize=73.80 KB heapSize=219.89 KB 2023-04-15 02:54:10,966 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK b7b06ed62cbe8eca9103c3d111564e6c, store=A 2023-04-15 02:54:10,966 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-15 02:54:10,966 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK b7b06ed62cbe8eca9103c3d111564e6c, store=B 2023-04-15 02:54:10,966 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-15 02:54:10,966 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK b7b06ed62cbe8eca9103c3d111564e6c, store=C 2023-04-15 02:54:10,967 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-15 02:54:11,008 DEBUG [Thread-864] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-15 02:54:11,008 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-15 02:54:11,010 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-15 02:54:11,011 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-15 02:54:11,011 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-15 02:54:11,011 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-15 02:54:11,011 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-15 02:54:11,011 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-15 02:54:11,011 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-15 02:54:11,011 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:11,011 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:54:11,012 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-15 02:54:11,012 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:11,012 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-15 02:54:11,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-15 02:54:11,013 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:11,013 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-15 02:54:11,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:54:11,013 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:11,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-15 02:54:11,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:54:11,013 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-15 02:54:11,014 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-15 02:54:11,014 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-15 02:54:11,014 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-15 02:54:11,014 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-15 02:54:11,014 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-15 02:54:11,014 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-15 02:54:11,014 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681527240182.b7b06ed62cbe8eca9103c3d111564e6c. 2023-04-15 02:54:11,014 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681527240182.b7b06ed62cbe8eca9103c3d111564e6c. started... 2023-04-15 02:54:11,014 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681527240182.b7b06ed62cbe8eca9103c3d111564e6c. as already flushing 2023-04-15 02:54:11,014 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681527240182.b7b06ed62cbe8eca9103c3d111564e6c. 2023-04-15 02:54:11,014 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-15 02:54:11,014 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-15 02:54:11,014 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:11,014 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-15 02:54:11,014 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34681,1681527209133' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-15 02:54:11,015 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:11,015 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:11,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:11,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:54:11,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:54:11,015 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:11,015 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-15 02:54:11,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:54:11,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:54:11,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:54:11,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:11,016 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b7b06ed62cbe8eca9103c3d111564e6c, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:54:11,016 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34681] ipc.CallRunner(144): callId: 114 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:39132 deadline: 1681527311012, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b7b06ed62cbe8eca9103c3d111564e6c, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:11,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:54:11,016 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34681,1681527209133' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-15 02:54:11,016 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@c5417e6[Count = 0] remaining members to acquire global barrier 2023-04-15 02:54:11,016 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-15 02:54:11,017 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:11,017 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:11,017 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:11,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:11,017 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-15 02:54:11,017 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:11,017 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-15 02:54:11,017 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-15 02:54:11,017 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34681,1681527209133' in zk 2023-04-15 02:54:11,018 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-15 02:54:11,018 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:11,018 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-15 02:54:11,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:11,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:54:11,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:54:11,018 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-15 02:54:11,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:54:11,018 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b7b06ed62cbe8eca9103c3d111564e6c, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:54:11,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:54:11,018 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34681] ipc.CallRunner(144): callId: 122 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:39106 deadline: 1681527311013, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b7b06ed62cbe8eca9103c3d111564e6c, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:11,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:54:11,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:11,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:54:11,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:54:11,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:11,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34681,1681527209133': 2023-04-15 02:54:11,020 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34681,1681527209133' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-15 02:54:11,020 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-15 02:54:11,020 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-15 02:54:11,020 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-15 02:54:11,020 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-15 02:54:11,020 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-15 02:54:11,021 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:11,021 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-15 02:54:11,021 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:11,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:11,021 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-15 02:54:11,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-15 02:54:11,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:11,022 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:11,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:11,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:54:11,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:54:11,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:54:11,023 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:11,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:54:11,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:54:11,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:54:11,023 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:11,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:11,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:54:11,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:54:11,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:11,025 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:11,025 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:54:11,025 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:54:11,025 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:11,025 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:11,025 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:11,025 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:11,025 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-15 02:54:11,025 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-15 02:54:11,025 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-15 02:54:11,025 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-15 02:54:11,025 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-15 02:54:11,025 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-15 02:54:11,025 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-15 02:54:11,025 DEBUG [Thread-864] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-15 02:54:11,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-15 02:54:11,025 DEBUG [Thread-864] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-15 02:54:11,026 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-15 02:54:11,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-15 02:54:11,119 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b7b06ed62cbe8eca9103c3d111564e6c, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:54:11,119 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34681] ipc.CallRunner(144): callId: 116 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:39132 deadline: 1681527311117, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b7b06ed62cbe8eca9103c3d111564e6c, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:11,121 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b7b06ed62cbe8eca9103c3d111564e6c, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:54:11,121 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34681] ipc.CallRunner(144): callId: 124 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:39106 deadline: 1681527311119, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b7b06ed62cbe8eca9103c3d111564e6c, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:11,126 DEBUG [Thread-864] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-15 02:54:11,126 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-15 02:54:11,127 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-15 02:54:11,128 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-15 02:54:11,128 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-15 02:54:11,128 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-15 02:54:11,128 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-15 02:54:11,128 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-15 02:54:11,128 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-15 02:54:11,129 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:11,129 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:54:11,129 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:11,129 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-15 02:54:11,129 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-15 02:54:11,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-15 02:54:11,130 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:11,130 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-15 02:54:11,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:54:11,130 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:11,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-15 02:54:11,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:54:11,130 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-15 02:54:11,130 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-15 02:54:11,130 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-15 02:54:11,131 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-15 02:54:11,131 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-15 02:54:11,131 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-15 02:54:11,131 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-15 02:54:11,131 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-15 02:54:11,131 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681527240182.b7b06ed62cbe8eca9103c3d111564e6c. 2023-04-15 02:54:11,131 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681527240182.b7b06ed62cbe8eca9103c3d111564e6c. started... 2023-04-15 02:54:11,131 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681527240182.b7b06ed62cbe8eca9103c3d111564e6c. as already flushing 2023-04-15 02:54:11,131 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681527240182.b7b06ed62cbe8eca9103c3d111564e6c. 2023-04-15 02:54:11,131 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-15 02:54:11,131 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-15 02:54:11,131 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:11,131 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-15 02:54:11,131 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34681,1681527209133' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-15 02:54:11,132 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:11,132 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:11,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:11,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:54:11,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:54:11,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:54:11,132 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:11,132 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-15 02:54:11,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:54:11,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:54:11,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:11,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:54:11,133 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34681,1681527209133' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-15 02:54:11,133 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7f4fd1a3[Count = 0] remaining members to acquire global barrier 2023-04-15 02:54:11,133 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-15 02:54:11,133 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:11,134 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:11,134 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:11,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:11,134 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-15 02:54:11,134 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-15 02:54:11,134 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34681,1681527209133' in zk 2023-04-15 02:54:11,134 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:11,134 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-15 02:54:11,135 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-15 02:54:11,135 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:11,135 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-15 02:54:11,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:11,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:54:11,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:54:11,135 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-15 02:54:11,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:54:11,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:54:11,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:54:11,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:11,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:54:11,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:54:11,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:11,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34681,1681527209133': 2023-04-15 02:54:11,137 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34681,1681527209133' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-15 02:54:11,137 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-15 02:54:11,137 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-15 02:54:11,137 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-15 02:54:11,137 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-15 02:54:11,137 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-15 02:54:11,138 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:11,138 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:11,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:11,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:54:11,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:54:11,138 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:11,138 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-15 02:54:11,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:11,138 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:11,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:54:11,138 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-15 02:54:11,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-15 02:54:11,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:11,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:54:11,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:54:11,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:54:11,140 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:11,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:11,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:54:11,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:54:11,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:11,141 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:11,141 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:54:11,141 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:54:11,141 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:11,141 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:11,141 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-15 02:54:11,141 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-15 02:54:11,141 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-15 02:54:11,141 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-15 02:54:11,141 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-15 02:54:11,141 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-15 02:54:11,141 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:11,141 DEBUG [Thread-864] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-15 02:54:11,141 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-15 02:54:11,141 DEBUG [Thread-864] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-15 02:54:11,141 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:11,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-15 02:54:11,142 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-15 02:54:11,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-15 02:54:11,161 DEBUG [RS:0;jenkins-hbase20:34681-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/b7b06ed62cbe8eca9103c3d111564e6c/.tmp/B/b72caa2df2b34bf69d99434a82ee9ac0 as hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/b7b06ed62cbe8eca9103c3d111564e6c/B/b72caa2df2b34bf69d99434a82ee9ac0 2023-04-15 02:54:11,167 INFO [RS:0;jenkins-hbase20:34681-longCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in b7b06ed62cbe8eca9103c3d111564e6c/B of b7b06ed62cbe8eca9103c3d111564e6c into b72caa2df2b34bf69d99434a82ee9ac0(size=12.0 K), total size for store is 23.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-15 02:54:11,168 DEBUG [RS:0;jenkins-hbase20:34681-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for b7b06ed62cbe8eca9103c3d111564e6c: 2023-04-15 02:54:11,168 INFO [RS:0;jenkins-hbase20:34681-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681527240182.b7b06ed62cbe8eca9103c3d111564e6c., storeName=b7b06ed62cbe8eca9103c3d111564e6c/B, priority=11, startTime=1681527250733; duration=0sec 2023-04-15 02:54:11,168 DEBUG [RS:0;jenkins-hbase20:34681-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-15 02:54:11,168 DEBUG [RS:0;jenkins-hbase20:34681-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 6 store files, 0 compacting, 6 eligible, 16 blocking 2023-04-15 02:54:11,170 DEBUG [RS:0;jenkins-hbase20:34681-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 6 files of size 69475 starting at candidate #0 after considering 10 permutations with 10 in ratio 2023-04-15 02:54:11,170 DEBUG [RS:0;jenkins-hbase20:34681-longCompactions-0] regionserver.HStore(1912): b7b06ed62cbe8eca9103c3d111564e6c/C is initiating minor compaction (all files) 2023-04-15 02:54:11,170 INFO [RS:0;jenkins-hbase20:34681-longCompactions-0] regionserver.HRegion(2259): Starting compaction of b7b06ed62cbe8eca9103c3d111564e6c/C in TestAcidGuarantees,,1681527240182.b7b06ed62cbe8eca9103c3d111564e6c. 2023-04-15 02:54:11,170 INFO [RS:0;jenkins-hbase20:34681-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/b7b06ed62cbe8eca9103c3d111564e6c/C/737e570c76d24414ac42ba9b0bd82ec0, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/b7b06ed62cbe8eca9103c3d111564e6c/C/d664f390a19847bf93af9bc550f95fda, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/b7b06ed62cbe8eca9103c3d111564e6c/C/8b5d68def4f844bcbbbb22de6f29644a, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/b7b06ed62cbe8eca9103c3d111564e6c/C/dccb7202a6a94ef58345640fce8cc84f, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/b7b06ed62cbe8eca9103c3d111564e6c/C/058660d60eb74ece847d2441e5be3182, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/b7b06ed62cbe8eca9103c3d111564e6c/C/be56c7c9f6194b33b06978f39ebd5f62] into tmpdir=hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/b7b06ed62cbe8eca9103c3d111564e6c/.tmp, totalSize=67.8 K 2023-04-15 02:54:11,171 DEBUG [RS:0;jenkins-hbase20:34681-longCompactions-0] compactions.Compactor(207): Compacting 737e570c76d24414ac42ba9b0bd82ec0, keycount=150, bloomtype=ROW, size=11.7 K, encoding=NONE, compression=NONE, seqNum=62, earliestPutTs=1681527244084 2023-04-15 02:54:11,171 DEBUG [RS:0;jenkins-hbase20:34681-longCompactions-0] compactions.Compactor(207): Compacting d664f390a19847bf93af9bc550f95fda, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=96, earliestPutTs=1681527244766 2023-04-15 02:54:11,172 DEBUG [RS:0;jenkins-hbase20:34681-longCompactions-0] compactions.Compactor(207): Compacting 8b5d68def4f844bcbbbb22de6f29644a, keycount=100, bloomtype=ROW, size=9.3 K, encoding=NONE, compression=NONE, seqNum=111, earliestPutTs=1681527245396 2023-04-15 02:54:11,172 DEBUG [RS:0;jenkins-hbase20:34681-longCompactions-0] compactions.Compactor(207): Compacting dccb7202a6a94ef58345640fce8cc84f, keycount=150, bloomtype=ROW, size=11.7 K, encoding=NONE, compression=NONE, seqNum=131, earliestPutTs=1681527246571 2023-04-15 02:54:11,172 DEBUG [RS:0;jenkins-hbase20:34681-longCompactions-0] compactions.Compactor(207): Compacting 058660d60eb74ece847d2441e5be3182, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=144, earliestPutTs=1681527248703 2023-04-15 02:54:11,173 DEBUG [RS:0;jenkins-hbase20:34681-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/b7b06ed62cbe8eca9103c3d111564e6c/.tmp/A/a9fb2169d5184cbe8e4682c53928ba72 as hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/b7b06ed62cbe8eca9103c3d111564e6c/A/a9fb2169d5184cbe8e4682c53928ba72 2023-04-15 02:54:11,173 DEBUG [RS:0;jenkins-hbase20:34681-longCompactions-0] compactions.Compactor(207): Compacting be56c7c9f6194b33b06978f39ebd5f62, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=163, earliestPutTs=1681527249855 2023-04-15 02:54:11,183 INFO [RS:0;jenkins-hbase20:34681-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in b7b06ed62cbe8eca9103c3d111564e6c/A of b7b06ed62cbe8eca9103c3d111564e6c into a9fb2169d5184cbe8e4682c53928ba72(size=25.6 K), total size for store is 50.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-15 02:54:11,183 DEBUG [RS:0;jenkins-hbase20:34681-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for b7b06ed62cbe8eca9103c3d111564e6c: 2023-04-15 02:54:11,183 INFO [RS:0;jenkins-hbase20:34681-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681527240182.b7b06ed62cbe8eca9103c3d111564e6c., storeName=b7b06ed62cbe8eca9103c3d111564e6c/A, priority=11, startTime=1681527250732; duration=0sec 2023-04-15 02:54:11,183 DEBUG [RS:0;jenkins-hbase20:34681-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-15 02:54:11,184 INFO [RS:0;jenkins-hbase20:34681-longCompactions-0] throttle.PressureAwareThroughputController(145): b7b06ed62cbe8eca9103c3d111564e6c#C#compaction#132 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-15 02:54:11,242 DEBUG [Thread-864] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-15 02:54:11,242 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-15 02:54:11,244 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-15 02:54:11,244 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-15 02:54:11,244 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-15 02:54:11,244 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-15 02:54:11,245 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-15 02:54:11,245 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-15 02:54:11,245 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-15 02:54:11,245 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:11,245 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:54:11,246 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:11,246 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-15 02:54:11,246 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-15 02:54:11,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-15 02:54:11,246 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:11,246 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-15 02:54:11,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:54:11,246 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:11,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-15 02:54:11,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:54:11,247 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-15 02:54:11,247 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-15 02:54:11,247 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-15 02:54:11,247 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-15 02:54:11,247 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-15 02:54:11,247 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-15 02:54:11,247 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-15 02:54:11,247 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-15 02:54:11,247 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681527240182.b7b06ed62cbe8eca9103c3d111564e6c. 2023-04-15 02:54:11,247 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681527240182.b7b06ed62cbe8eca9103c3d111564e6c. started... 2023-04-15 02:54:11,247 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681527240182.b7b06ed62cbe8eca9103c3d111564e6c. as already flushing 2023-04-15 02:54:11,247 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681527240182.b7b06ed62cbe8eca9103c3d111564e6c. 2023-04-15 02:54:11,247 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-15 02:54:11,247 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-15 02:54:11,247 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:11,247 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-15 02:54:11,247 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34681,1681527209133' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-15 02:54:11,248 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:11,248 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:11,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:11,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:54:11,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:54:11,248 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:11,248 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-15 02:54:11,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:54:11,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:54:11,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:54:11,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:11,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:54:11,249 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34681,1681527209133' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-15 02:54:11,249 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@35fcaa46[Count = 0] remaining members to acquire global barrier 2023-04-15 02:54:11,249 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-15 02:54:11,249 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:11,250 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:11,250 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:11,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:11,250 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-15 02:54:11,250 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-15 02:54:11,250 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34681,1681527209133' in zk 2023-04-15 02:54:11,250 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:11,250 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-15 02:54:11,251 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-15 02:54:11,251 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:11,251 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-15 02:54:11,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:11,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:54:11,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:54:11,251 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-15 02:54:11,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:54:11,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:54:11,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:54:11,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:11,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:54:11,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:54:11,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:11,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34681,1681527209133': 2023-04-15 02:54:11,253 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34681,1681527209133' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-15 02:54:11,253 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-15 02:54:11,253 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-15 02:54:11,253 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-15 02:54:11,253 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-15 02:54:11,253 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-15 02:54:11,254 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:11,254 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:11,254 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-15 02:54:11,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:11,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:54:11,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:54:11,254 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:11,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:11,255 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:11,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:54:11,255 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-15 02:54:11,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-15 02:54:11,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:11,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:54:11,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:54:11,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:54:11,256 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:11,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:11,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:54:11,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:54:11,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:11,257 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:11,257 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:54:11,257 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:54:11,257 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:11,257 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-15 02:54:11,257 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-15 02:54:11,257 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-15 02:54:11,257 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-15 02:54:11,257 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-15 02:54:11,257 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:11,257 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-15 02:54:11,257 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-15 02:54:11,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-15 02:54:11,257 DEBUG [Thread-864] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-15 02:54:11,257 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:11,257 DEBUG [Thread-864] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-15 02:54:11,257 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:11,257 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-15 02:54:11,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-15 02:54:11,321 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b7b06ed62cbe8eca9103c3d111564e6c, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:54:11,321 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34681] ipc.CallRunner(144): callId: 118 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:39132 deadline: 1681527311320, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b7b06ed62cbe8eca9103c3d111564e6c, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:11,323 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b7b06ed62cbe8eca9103c3d111564e6c, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:54:11,323 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34681] ipc.CallRunner(144): callId: 126 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:39106 deadline: 1681527311322, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b7b06ed62cbe8eca9103c3d111564e6c, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:11,357 DEBUG [Thread-864] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-15 02:54:11,358 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-15 02:54:11,359 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-15 02:54:11,360 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-15 02:54:11,360 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-15 02:54:11,360 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-15 02:54:11,360 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-15 02:54:11,360 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-15 02:54:11,360 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-15 02:54:11,361 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:11,361 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:54:11,361 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-15 02:54:11,361 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:11,362 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-15 02:54:11,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-15 02:54:11,362 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:11,362 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-15 02:54:11,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:54:11,362 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:11,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-15 02:54:11,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:54:11,362 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-15 02:54:11,363 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-15 02:54:11,363 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-15 02:54:11,363 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-15 02:54:11,363 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-15 02:54:11,363 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-15 02:54:11,363 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-15 02:54:11,363 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-15 02:54:11,363 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681527240182.b7b06ed62cbe8eca9103c3d111564e6c. 2023-04-15 02:54:11,363 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681527240182.b7b06ed62cbe8eca9103c3d111564e6c. started... 2023-04-15 02:54:11,363 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681527240182.b7b06ed62cbe8eca9103c3d111564e6c. as already flushing 2023-04-15 02:54:11,363 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681527240182.b7b06ed62cbe8eca9103c3d111564e6c. 2023-04-15 02:54:11,363 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-15 02:54:11,363 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-15 02:54:11,363 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:11,363 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-15 02:54:11,363 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34681,1681527209133' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-15 02:54:11,364 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:11,364 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:11,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:11,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:54:11,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:54:11,365 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:11,365 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-15 02:54:11,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:54:11,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:54:11,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:54:11,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:11,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:54:11,366 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34681,1681527209133' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-15 02:54:11,366 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6234236d[Count = 0] remaining members to acquire global barrier 2023-04-15 02:54:11,366 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-15 02:54:11,366 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:11,366 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:11,367 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:11,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:11,367 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-15 02:54:11,367 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-15 02:54:11,367 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34681,1681527209133' in zk 2023-04-15 02:54:11,367 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:11,367 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-15 02:54:11,367 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:11,367 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-15 02:54:11,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:11,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:54:11,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:54:11,367 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-15 02:54:11,367 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-15 02:54:11,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:54:11,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:54:11,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:54:11,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:11,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:54:11,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:54:11,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:11,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34681,1681527209133': 2023-04-15 02:54:11,369 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34681,1681527209133' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-15 02:54:11,369 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-15 02:54:11,370 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-15 02:54:11,370 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-15 02:54:11,370 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-15 02:54:11,370 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-15 02:54:11,370 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:11,371 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-15 02:54:11,371 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:11,371 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:11,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:11,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:54:11,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:54:11,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:11,371 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-15 02:54:11,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-15 02:54:11,371 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:11,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:54:11,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:11,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:54:11,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:54:11,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:54:11,372 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:11,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:11,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:54:11,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:54:11,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:11,373 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:11,373 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:54:11,374 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:54:11,374 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:11,374 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:11,374 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:11,374 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:11,374 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-15 02:54:11,374 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-15 02:54:11,374 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-15 02:54:11,374 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-15 02:54:11,374 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-15 02:54:11,374 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-15 02:54:11,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-15 02:54:11,374 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-15 02:54:11,374 DEBUG [Thread-864] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-15 02:54:11,374 DEBUG [Thread-864] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-15 02:54:11,374 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-15 02:54:11,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-15 02:54:11,392 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023041533969fcf9d9a442996707e7dfe09d3af to hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023041533969fcf9d9a442996707e7dfe09d3af 2023-04-15 02:54:11,474 DEBUG [Thread-864] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-15 02:54:11,475 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-15 02:54:11,476 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-15 02:54:11,477 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-15 02:54:11,477 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-15 02:54:11,477 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-15 02:54:11,477 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-15 02:54:11,477 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-15 02:54:11,477 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-15 02:54:11,477 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:11,477 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:54:11,478 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-15 02:54:11,478 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-15 02:54:11,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-15 02:54:11,478 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:11,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:54:11,479 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:11,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-15 02:54:11,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:54:11,479 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-15 02:54:11,479 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-15 02:54:11,479 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-15 02:54:11,479 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-15 02:54:11,479 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-15 02:54:11,479 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-15 02:54:11,480 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-15 02:54:11,480 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-15 02:54:11,480 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681527240182.b7b06ed62cbe8eca9103c3d111564e6c. 2023-04-15 02:54:11,480 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681527240182.b7b06ed62cbe8eca9103c3d111564e6c. started... 2023-04-15 02:54:11,480 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681527240182.b7b06ed62cbe8eca9103c3d111564e6c. as already flushing 2023-04-15 02:54:11,480 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681527240182.b7b06ed62cbe8eca9103c3d111564e6c. 2023-04-15 02:54:11,480 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-15 02:54:11,480 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-15 02:54:11,480 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:11,480 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-15 02:54:11,480 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34681,1681527209133' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-15 02:54:11,481 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:11,481 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:11,481 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-15 02:54:11,482 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:11,482 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34681,1681527209133' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-15 02:54:11,482 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@237b31e0[Count = 0] remaining members to acquire global barrier 2023-04-15 02:54:11,482 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-15 02:54:11,482 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-15 02:54:11,482 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:11,483 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:11,483 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:11,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:11,483 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:11,483 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-15 02:54:11,483 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-15 02:54:11,483 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-15 02:54:11,483 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34681,1681527209133' in zk 2023-04-15 02:54:11,484 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:11,484 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-15 02:54:11,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:11,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:54:11,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:54:11,484 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-15 02:54:11,484 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-15 02:54:11,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:54:11,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:54:11,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:54:11,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:11,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:54:11,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:54:11,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:11,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34681,1681527209133': 2023-04-15 02:54:11,485 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34681,1681527209133' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-15 02:54:11,485 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-15 02:54:11,485 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-15 02:54:11,485 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-15 02:54:11,485 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-15 02:54:11,485 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-15 02:54:11,486 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:11,486 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-15 02:54:11,486 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:11,486 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:11,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:11,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:54:11,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:54:11,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:11,487 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:11,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:54:11,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:54:11,487 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-15 02:54:11,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-15 02:54:11,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:54:11,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:11,487 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:11,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:54:11,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:11,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:54:11,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:54:11,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:11,489 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:11,489 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:54:11,489 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:54:11,489 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-15 02:54:11,489 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-15 02:54:11,489 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-15 02:54:11,489 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-15 02:54:11,489 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:11,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-15 02:54:11,489 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-15 02:54:11,489 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-15 02:54:11,489 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-15 02:54:11,489 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:11,489 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:11,489 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:11,489 DEBUG [Thread-864] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-15 02:54:11,489 DEBUG [Thread-864] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-15 02:54:11,489 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-15 02:54:11,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-15 02:54:11,589 DEBUG [Thread-864] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-15 02:54:11,590 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-15 02:54:11,591 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-15 02:54:11,592 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-15 02:54:11,592 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-15 02:54:11,592 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-15 02:54:11,592 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-15 02:54:11,592 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-15 02:54:11,592 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-15 02:54:11,592 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:11,592 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:54:11,593 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-15 02:54:11,593 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:11,593 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-15 02:54:11,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-15 02:54:11,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:54:11,596 DEBUG [RS:0;jenkins-hbase20:34681-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/b7b06ed62cbe8eca9103c3d111564e6c/.tmp/C/a1fdb8330b4f401484a795f4913c211b as hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/b7b06ed62cbe8eca9103c3d111564e6c/C/a1fdb8330b4f401484a795f4913c211b 2023-04-15 02:54:11,604 INFO [RS:0;jenkins-hbase20:34681-longCompactions-0] regionserver.HStore(1652): Completed compaction of 6 (all) file(s) in b7b06ed62cbe8eca9103c3d111564e6c/C of b7b06ed62cbe8eca9103c3d111564e6c into a1fdb8330b4f401484a795f4913c211b(size=12.1 K), total size for store is 12.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-15 02:54:11,604 DEBUG [RS:0;jenkins-hbase20:34681-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for b7b06ed62cbe8eca9103c3d111564e6c: 2023-04-15 02:54:11,604 INFO [RS:0;jenkins-hbase20:34681-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681527240182.b7b06ed62cbe8eca9103c3d111564e6c., storeName=b7b06ed62cbe8eca9103c3d111564e6c/C, priority=10, startTime=1681527250733; duration=0sec 2023-04-15 02:54:11,604 DEBUG [RS:0;jenkins-hbase20:34681-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-15 02:54:11,606 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:11,606 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:11,606 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-15 02:54:11,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-15 02:54:11,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:54:11,606 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-15 02:54:11,616 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-15 02:54:11,616 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-15 02:54:11,616 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-15 02:54:11,616 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-15 02:54:11,616 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-15 02:54:11,616 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-15 02:54:11,616 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-15 02:54:11,616 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681527240182.b7b06ed62cbe8eca9103c3d111564e6c. 2023-04-15 02:54:11,616 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681527240182.b7b06ed62cbe8eca9103c3d111564e6c. started... 2023-04-15 02:54:11,616 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681527240182.b7b06ed62cbe8eca9103c3d111564e6c. as already flushing 2023-04-15 02:54:11,616 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681527240182.b7b06ed62cbe8eca9103c3d111564e6c. 2023-04-15 02:54:11,616 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-15 02:54:11,617 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-15 02:54:11,617 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:11,617 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-15 02:54:11,617 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34681,1681527209133' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-15 02:54:11,618 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:11,618 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:11,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:11,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:54:11,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:54:11,618 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:11,618 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-15 02:54:11,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:54:11,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:54:11,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:54:11,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:11,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:54:11,619 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34681,1681527209133' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-15 02:54:11,619 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4e73e3ea[Count = 0] remaining members to acquire global barrier 2023-04-15 02:54:11,619 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-15 02:54:11,619 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:11,620 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:11,620 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:11,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:11,620 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-15 02:54:11,620 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-15 02:54:11,620 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34681,1681527209133' in zk 2023-04-15 02:54:11,620 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:11,620 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-15 02:54:11,621 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-15 02:54:11,621 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:11,621 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-15 02:54:11,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:11,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:54:11,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:54:11,621 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-15 02:54:11,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:54:11,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:54:11,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:54:11,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:11,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:54:11,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:54:11,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:11,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34681,1681527209133': 2023-04-15 02:54:11,623 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34681,1681527209133' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-15 02:54:11,623 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-15 02:54:11,623 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-15 02:54:11,623 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-15 02:54:11,623 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-15 02:54:11,623 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-15 02:54:11,623 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b7b06ed62cbe8eca9103c3d111564e6c, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:54:11,623 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34681] ipc.CallRunner(144): callId: 120 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:39132 deadline: 1681527311622, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b7b06ed62cbe8eca9103c3d111564e6c, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:11,625 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:11,625 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:11,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:11,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:54:11,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:54:11,625 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:11,625 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-15 02:54:11,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:11,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:54:11,625 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-15 02:54:11,625 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:11,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:54:11,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-15 02:54:11,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:11,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:54:11,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:54:11,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:11,627 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:11,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:54:11,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:54:11,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:11,628 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-15 02:54:11,628 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-15 02:54:11,628 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:11,628 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-15 02:54:11,628 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:54:11,628 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:54:11,628 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-15 02:54:11,628 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-15 02:54:11,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-15 02:54:11,628 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:11,628 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-15 02:54:11,629 DEBUG [Thread-864] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-15 02:54:11,629 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:11,629 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:11,629 DEBUG [Thread-864] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-15 02:54:11,629 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:11,629 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-15 02:54:11,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-15 02:54:11,629 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b7b06ed62cbe8eca9103c3d111564e6c, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:54:11,629 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34681] ipc.CallRunner(144): callId: 128 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:39106 deadline: 1681527311626, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b7b06ed62cbe8eca9103c3d111564e6c, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:11,651 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b7b06ed62cbe8eca9103c3d111564e6c, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:54:11,651 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34681] ipc.CallRunner(144): callId: 23 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:39142 deadline: 1681527311650, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b7b06ed62cbe8eca9103c3d111564e6c, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:11,652 DEBUG [Thread-856] client.RpcRetryingCallerImpl(129): Call exception, tries=7, retries=16, started=8169 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b7b06ed62cbe8eca9103c3d111564e6c, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_1' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1681527240182.b7b06ed62cbe8eca9103c3d111564e6c., hostname=jenkins-hbase20.apache.org,34681,1681527209133, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b7b06ed62cbe8eca9103c3d111564e6c, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor31.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b7b06ed62cbe8eca9103c3d111564e6c, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-15 02:54:11,656 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b7b06ed62cbe8eca9103c3d111564e6c, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:54:11,656 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34681] ipc.CallRunner(144): callId: 26 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:39108 deadline: 1681527311653, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b7b06ed62cbe8eca9103c3d111564e6c, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:11,658 DEBUG [Thread-854] client.RpcRetryingCallerImpl(129): Call exception, tries=7, retries=16, started=8178 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b7b06ed62cbe8eca9103c3d111564e6c, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_0' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1681527240182.b7b06ed62cbe8eca9103c3d111564e6c., hostname=jenkins-hbase20.apache.org,34681,1681527209133, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b7b06ed62cbe8eca9103c3d111564e6c, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor31.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b7b06ed62cbe8eca9103c3d111564e6c, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-15 02:54:11,658 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b7b06ed62cbe8eca9103c3d111564e6c, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:54:11,658 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34681] ipc.CallRunner(144): callId: 25 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:39116 deadline: 1681527311656, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b7b06ed62cbe8eca9103c3d111564e6c, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:11,659 DEBUG [Thread-858] client.RpcRetryingCallerImpl(129): Call exception, tries=7, retries=16, started=8182 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b7b06ed62cbe8eca9103c3d111564e6c, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_0' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1681527240182.b7b06ed62cbe8eca9103c3d111564e6c., hostname=jenkins-hbase20.apache.org,34681,1681527209133, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b7b06ed62cbe8eca9103c3d111564e6c, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor31.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b7b06ed62cbe8eca9103c3d111564e6c, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-15 02:54:11,729 DEBUG [Thread-864] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-15 02:54:11,729 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-15 02:54:11,731 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-15 02:54:11,731 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-15 02:54:11,731 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-15 02:54:11,731 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-15 02:54:11,731 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-15 02:54:11,731 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-15 02:54:11,731 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-15 02:54:11,732 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:11,732 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:54:11,732 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:11,732 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-15 02:54:11,732 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-15 02:54:11,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-15 02:54:11,733 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:11,733 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-15 02:54:11,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:54:11,733 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:11,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-15 02:54:11,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:54:11,733 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-15 02:54:11,733 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-15 02:54:11,733 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-15 02:54:11,733 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-15 02:54:11,733 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-15 02:54:11,733 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-15 02:54:11,734 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-15 02:54:11,734 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-15 02:54:11,734 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681527240182.b7b06ed62cbe8eca9103c3d111564e6c. 2023-04-15 02:54:11,734 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681527240182.b7b06ed62cbe8eca9103c3d111564e6c. started... 2023-04-15 02:54:11,734 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681527240182.b7b06ed62cbe8eca9103c3d111564e6c. as already flushing 2023-04-15 02:54:11,734 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681527240182.b7b06ed62cbe8eca9103c3d111564e6c. 2023-04-15 02:54:11,734 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-15 02:54:11,734 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-15 02:54:11,734 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:11,734 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-15 02:54:11,734 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34681,1681527209133' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-15 02:54:11,735 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:11,735 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:11,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:11,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:54:11,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:54:11,735 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:11,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:54:11,735 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-15 02:54:11,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:54:11,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:54:11,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:11,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:54:11,736 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34681,1681527209133' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-15 02:54:11,736 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4f30f14e[Count = 0] remaining members to acquire global barrier 2023-04-15 02:54:11,736 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-15 02:54:11,736 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:11,736 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:11,736 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:11,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:11,737 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-15 02:54:11,737 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-15 02:54:11,737 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:11,737 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-15 02:54:11,737 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34681,1681527209133' in zk 2023-04-15 02:54:11,737 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:11,737 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-15 02:54:11,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:11,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:54:11,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:54:11,737 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-15 02:54:11,737 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-15 02:54:11,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:54:11,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:54:11,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:54:11,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:11,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:54:11,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:54:11,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:11,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34681,1681527209133': 2023-04-15 02:54:11,739 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34681,1681527209133' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-15 02:54:11,739 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-15 02:54:11,739 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-15 02:54:11,739 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-15 02:54:11,739 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-15 02:54:11,739 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-15 02:54:11,740 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:11,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:11,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:54:11,740 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:11,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:54:11,740 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:11,740 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-15 02:54:11,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:11,740 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-15 02:54:11,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-15 02:54:11,740 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:11,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:54:11,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:11,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:54:11,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:54:11,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:54:11,741 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:11,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:11,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:54:11,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:54:11,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:11,743 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-15 02:54:11,743 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-15 02:54:11,743 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:11,743 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:54:11,743 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:54:11,743 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-15 02:54:11,743 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-15 02:54:11,743 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-15 02:54:11,743 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-15 02:54:11,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-15 02:54:11,743 DEBUG [Thread-864] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-15 02:54:11,743 DEBUG [Thread-864] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-15 02:54:11,743 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:11,744 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:11,743 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-15 02:54:11,744 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:11,744 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:11,744 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-15 02:54:11,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-15 02:54:11,797 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=177, memsize=24.6 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/b7b06ed62cbe8eca9103c3d111564e6c/.tmp/A/f062d2172d864cc7aac2415982fffc57 2023-04-15 02:54:11,844 DEBUG [Thread-864] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-15 02:54:11,844 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-15 02:54:11,845 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-15 02:54:11,846 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-15 02:54:11,846 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-15 02:54:11,846 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-15 02:54:11,846 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-15 02:54:11,846 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-15 02:54:11,846 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-15 02:54:11,847 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:11,847 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:54:11,847 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-15 02:54:11,848 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-15 02:54:11,848 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:11,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-15 02:54:11,848 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:11,848 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-15 02:54:11,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:54:11,848 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:11,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-15 02:54:11,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:54:11,849 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-15 02:54:11,849 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-15 02:54:11,849 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-15 02:54:11,849 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-15 02:54:11,849 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-15 02:54:11,849 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-15 02:54:11,849 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-15 02:54:11,849 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-15 02:54:11,849 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681527240182.b7b06ed62cbe8eca9103c3d111564e6c. 2023-04-15 02:54:11,849 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681527240182.b7b06ed62cbe8eca9103c3d111564e6c. started... 2023-04-15 02:54:11,849 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681527240182.b7b06ed62cbe8eca9103c3d111564e6c. as already flushing 2023-04-15 02:54:11,849 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681527240182.b7b06ed62cbe8eca9103c3d111564e6c. 2023-04-15 02:54:11,849 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-15 02:54:11,849 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-15 02:54:11,849 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:11,849 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-15 02:54:11,849 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34681,1681527209133' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-15 02:54:11,850 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:11,850 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:11,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:11,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:54:11,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:54:11,851 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:11,851 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-15 02:54:11,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:54:11,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:54:11,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:54:11,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:11,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:54:11,852 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34681,1681527209133' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-15 02:54:11,852 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1f4ecc74[Count = 0] remaining members to acquire global barrier 2023-04-15 02:54:11,852 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-15 02:54:11,852 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:11,852 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:11,852 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:11,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:11,852 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-15 02:54:11,852 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-15 02:54:11,852 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34681,1681527209133' in zk 2023-04-15 02:54:11,853 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:11,853 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-15 02:54:11,853 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:11,853 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-15 02:54:11,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:11,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:54:11,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:54:11,853 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-15 02:54:11,853 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-15 02:54:11,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:54:11,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:54:11,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:54:11,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:11,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:54:11,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:54:11,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:11,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34681,1681527209133': 2023-04-15 02:54:11,855 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34681,1681527209133' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-15 02:54:11,855 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-15 02:54:11,855 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-15 02:54:11,855 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-15 02:54:11,855 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-15 02:54:11,855 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-15 02:54:11,856 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:11,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:11,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:54:11,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:54:11,856 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:11,857 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-15 02:54:11,857 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:11,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:11,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:54:11,857 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-15 02:54:11,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-15 02:54:11,857 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:11,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:11,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:54:11,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:54:11,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:54:11,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:11,858 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:11,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:54:11,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:54:11,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:11,860 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-15 02:54:11,860 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-15 02:54:11,860 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-15 02:54:11,860 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:11,860 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-15 02:54:11,860 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-15 02:54:11,861 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-15 02:54:11,860 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:54:11,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-15 02:54:11,860 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-15 02:54:11,861 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:54:11,861 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:11,861 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:11,861 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:11,861 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:11,861 DEBUG [Thread-864] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-15 02:54:11,861 DEBUG [Thread-864] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-15 02:54:11,861 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-15 02:54:11,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-15 02:54:11,961 DEBUG [Thread-864] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-15 02:54:11,962 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-15 02:54:11,963 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-15 02:54:11,964 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-15 02:54:11,964 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-15 02:54:11,964 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-15 02:54:11,964 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-15 02:54:11,964 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-15 02:54:11,964 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-15 02:54:11,964 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:11,964 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:54:11,965 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-15 02:54:11,965 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-15 02:54:11,965 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:11,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-15 02:54:11,965 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:11,965 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-15 02:54:11,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:54:11,966 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:11,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-15 02:54:11,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:54:11,966 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-15 02:54:11,966 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-15 02:54:11,966 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-15 02:54:11,966 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-15 02:54:11,966 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-15 02:54:11,966 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-15 02:54:11,967 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-15 02:54:11,967 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-15 02:54:11,967 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681527240182.b7b06ed62cbe8eca9103c3d111564e6c. 2023-04-15 02:54:11,967 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681527240182.b7b06ed62cbe8eca9103c3d111564e6c. started... 2023-04-15 02:54:11,967 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681527240182.b7b06ed62cbe8eca9103c3d111564e6c. as already flushing 2023-04-15 02:54:11,967 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681527240182.b7b06ed62cbe8eca9103c3d111564e6c. 2023-04-15 02:54:11,967 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-15 02:54:11,967 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-15 02:54:11,967 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:11,967 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-15 02:54:11,968 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34681,1681527209133' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-15 02:54:11,968 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:11,968 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:11,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:11,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:54:11,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:54:11,969 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:11,969 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-15 02:54:11,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:54:11,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:54:11,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:54:11,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:11,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:54:11,970 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34681,1681527209133' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-15 02:54:11,970 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5ca2bbbb[Count = 0] remaining members to acquire global barrier 2023-04-15 02:54:11,970 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-15 02:54:11,970 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:11,970 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:11,970 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:11,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:11,970 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-15 02:54:11,970 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-15 02:54:11,970 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:11,971 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-15 02:54:11,970 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34681,1681527209133' in zk 2023-04-15 02:54:11,971 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:11,971 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-15 02:54:11,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:11,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:54:11,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:54:11,971 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-15 02:54:11,971 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-15 02:54:11,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:54:11,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:54:11,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:54:11,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:11,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:54:11,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:54:11,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:11,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34681,1681527209133': 2023-04-15 02:54:11,973 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34681,1681527209133' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-15 02:54:11,973 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-15 02:54:11,973 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-15 02:54:11,973 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-15 02:54:11,973 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-15 02:54:11,973 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-15 02:54:11,974 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:11,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:11,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:54:11,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:54:11,974 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:11,974 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-15 02:54:11,974 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:11,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:11,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:54:11,974 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:11,974 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-15 02:54:11,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-15 02:54:11,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:54:11,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:11,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:54:11,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:54:11,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:11,975 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:11,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:54:11,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:54:11,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:11,977 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:11,977 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:54:11,977 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:54:11,977 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-15 02:54:11,977 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-15 02:54:11,977 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-15 02:54:11,977 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-15 02:54:11,977 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:11,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-15 02:54:11,977 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-15 02:54:11,977 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-15 02:54:11,977 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-15 02:54:11,977 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:11,977 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:11,977 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:11,978 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-15 02:54:11,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-15 02:54:11,978 DEBUG [Thread-864] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-15 02:54:11,978 DEBUG [Thread-864] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-15 02:54:12,078 DEBUG [Thread-864] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-15 02:54:12,078 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-15 02:54:12,080 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-15 02:54:12,081 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-15 02:54:12,081 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-15 02:54:12,081 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-15 02:54:12,081 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-15 02:54:12,081 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-15 02:54:12,081 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-15 02:54:12,081 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:12,081 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:54:12,082 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-15 02:54:12,082 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:12,082 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-15 02:54:12,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-15 02:54:12,082 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:12,082 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-15 02:54:12,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:54:12,082 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:12,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-15 02:54:12,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:54:12,083 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-15 02:54:12,083 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-15 02:54:12,083 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-15 02:54:12,083 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-15 02:54:12,083 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-15 02:54:12,083 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-15 02:54:12,083 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-15 02:54:12,083 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-15 02:54:12,083 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681527240182.b7b06ed62cbe8eca9103c3d111564e6c. 2023-04-15 02:54:12,083 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681527240182.b7b06ed62cbe8eca9103c3d111564e6c. started... 2023-04-15 02:54:12,083 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681527240182.b7b06ed62cbe8eca9103c3d111564e6c. as already flushing 2023-04-15 02:54:12,083 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681527240182.b7b06ed62cbe8eca9103c3d111564e6c. 2023-04-15 02:54:12,083 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-15 02:54:12,083 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-15 02:54:12,084 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:12,084 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-15 02:54:12,084 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34681,1681527209133' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-15 02:54:12,084 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:12,084 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:12,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:12,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:54:12,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:54:12,085 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:12,085 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-15 02:54:12,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:54:12,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:54:12,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:54:12,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:12,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:54:12,086 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34681,1681527209133' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-15 02:54:12,086 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@d9f6f76[Count = 0] remaining members to acquire global barrier 2023-04-15 02:54:12,086 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-15 02:54:12,086 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:12,086 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:12,086 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:12,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:12,087 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-15 02:54:12,087 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-15 02:54:12,087 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34681,1681527209133' in zk 2023-04-15 02:54:12,087 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:12,087 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-15 02:54:12,087 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:12,087 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-15 02:54:12,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:12,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:54:12,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:54:12,087 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-15 02:54:12,088 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-15 02:54:12,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:54:12,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:54:12,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:54:12,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:12,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:54:12,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:54:12,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:12,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34681,1681527209133': 2023-04-15 02:54:12,089 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34681,1681527209133' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-15 02:54:12,089 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-15 02:54:12,089 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-15 02:54:12,089 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-15 02:54:12,089 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-15 02:54:12,089 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-15 02:54:12,090 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:12,090 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:12,090 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-15 02:54:12,090 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:12,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:12,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:54:12,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:54:12,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:12,091 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-15 02:54:12,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-15 02:54:12,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:54:12,091 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:12,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:12,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:54:12,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:54:12,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:54:12,102 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:12,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:12,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:54:12,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:54:12,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:12,112 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:12,112 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-15 02:54:12,112 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:54:12,113 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-15 02:54:12,112 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-15 02:54:12,112 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-15 02:54:12,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-15 02:54:12,113 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:54:12,113 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-15 02:54:12,113 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-15 02:54:12,113 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:12,113 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:12,113 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:12,113 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:12,113 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-15 02:54:12,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-15 02:54:12,113 DEBUG [Thread-864] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-15 02:54:12,113 DEBUG [Thread-864] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-15 02:54:12,125 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b7b06ed62cbe8eca9103c3d111564e6c, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:54:12,125 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34681] ipc.CallRunner(144): callId: 122 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:39132 deadline: 1681527312124, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b7b06ed62cbe8eca9103c3d111564e6c, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:12,131 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b7b06ed62cbe8eca9103c3d111564e6c, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:54:12,132 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34681] ipc.CallRunner(144): callId: 130 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:39106 deadline: 1681527312131, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b7b06ed62cbe8eca9103c3d111564e6c, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:12,208 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=177 (bloomFilter=true), to=hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/b7b06ed62cbe8eca9103c3d111564e6c/.tmp/B/9b76848f28af4f69ae6328d74ae603f8 2023-04-15 02:54:12,213 DEBUG [Thread-864] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-15 02:54:12,214 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-15 02:54:12,216 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-15 02:54:12,230 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-15 02:54:12,230 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-15 02:54:12,230 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-15 02:54:12,230 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-15 02:54:12,230 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-15 02:54:12,230 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-15 02:54:12,231 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:12,231 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:54:12,231 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:12,231 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-15 02:54:12,232 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-15 02:54:12,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-15 02:54:12,232 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:12,232 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-15 02:54:12,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:54:12,232 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:12,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-15 02:54:12,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:54:12,232 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-15 02:54:12,234 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-15 02:54:12,234 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-15 02:54:12,234 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-15 02:54:12,234 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-15 02:54:12,234 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-15 02:54:12,244 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-15 02:54:12,244 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-15 02:54:12,255 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681527240182.b7b06ed62cbe8eca9103c3d111564e6c. 2023-04-15 02:54:12,255 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681527240182.b7b06ed62cbe8eca9103c3d111564e6c. started... 2023-04-15 02:54:12,255 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681527240182.b7b06ed62cbe8eca9103c3d111564e6c. as already flushing 2023-04-15 02:54:12,255 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681527240182.b7b06ed62cbe8eca9103c3d111564e6c. 2023-04-15 02:54:12,255 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-15 02:54:12,255 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-15 02:54:12,255 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:12,255 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-15 02:54:12,255 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34681,1681527209133' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-15 02:54:12,256 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=177 (bloomFilter=true), to=hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/b7b06ed62cbe8eca9103c3d111564e6c/.tmp/C/721c2abb2f524608bf5f421c3d3f9656 2023-04-15 02:54:12,257 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:12,257 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:12,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:12,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:54:12,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:54:12,258 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:12,258 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-15 02:54:12,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:54:12,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:54:12,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:54:12,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:12,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:54:12,260 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34681,1681527209133' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-15 02:54:12,261 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@45ed550a[Count = 0] remaining members to acquire global barrier 2023-04-15 02:54:12,261 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-15 02:54:12,261 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:12,262 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/b7b06ed62cbe8eca9103c3d111564e6c/.tmp/A/f062d2172d864cc7aac2415982fffc57 as hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/b7b06ed62cbe8eca9103c3d111564e6c/A/f062d2172d864cc7aac2415982fffc57 2023-04-15 02:54:12,267 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:12,267 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/b7b06ed62cbe8eca9103c3d111564e6c/A/f062d2172d864cc7aac2415982fffc57, entries=200, sequenceid=177, filesize=32.1 K 2023-04-15 02:54:12,267 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:12,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:12,267 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-15 02:54:12,267 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-15 02:54:12,267 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34681,1681527209133' in zk 2023-04-15 02:54:12,267 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:12,267 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-15 02:54:12,268 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/b7b06ed62cbe8eca9103c3d111564e6c/.tmp/B/9b76848f28af4f69ae6328d74ae603f8 as hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/b7b06ed62cbe8eca9103c3d111564e6c/B/9b76848f28af4f69ae6328d74ae603f8 2023-04-15 02:54:12,268 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-15 02:54:12,268 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-15 02:54:12,268 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-15 02:54:12,268 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:12,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:12,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:54:12,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:54:12,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:54:12,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:54:12,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:54:12,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:12,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:54:12,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:54:12,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:12,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34681,1681527209133': 2023-04-15 02:54:12,272 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34681,1681527209133' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-15 02:54:12,272 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-15 02:54:12,272 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-15 02:54:12,272 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-15 02:54:12,272 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-15 02:54:12,272 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-15 02:54:12,273 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/b7b06ed62cbe8eca9103c3d111564e6c/B/9b76848f28af4f69ae6328d74ae603f8, entries=150, sequenceid=177, filesize=11.8 K 2023-04-15 02:54:12,274 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/b7b06ed62cbe8eca9103c3d111564e6c/.tmp/C/721c2abb2f524608bf5f421c3d3f9656 as hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/b7b06ed62cbe8eca9103c3d111564e6c/C/721c2abb2f524608bf5f421c3d3f9656 2023-04-15 02:54:12,274 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:12,274 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:12,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:12,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:54:12,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:54:12,274 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:12,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:12,274 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-15 02:54:12,275 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-15 02:54:12,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-15 02:54:12,275 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:12,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:12,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:54:12,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:54:12,277 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:12,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:54:12,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:54:12,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:12,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:54:12,280 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:12,280 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:54:12,280 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-15 02:54:12,280 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-15 02:54:12,280 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-15 02:54:12,280 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:54:12,280 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-15 02:54:12,280 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:12,280 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-15 02:54:12,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-15 02:54:12,280 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:12,280 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/b7b06ed62cbe8eca9103c3d111564e6c/C/721c2abb2f524608bf5f421c3d3f9656, entries=150, sequenceid=177, filesize=11.8 K 2023-04-15 02:54:12,280 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-15 02:54:12,280 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:12,280 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:12,280 DEBUG [Thread-864] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-15 02:54:12,281 DEBUG [Thread-864] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-15 02:54:12,281 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-15 02:54:12,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-15 02:54:12,281 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-15 02:54:12,282 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~73.80 KB/75570, heapSize ~219.84 KB/225120, currentSize=100.63 KB/103050 for b7b06ed62cbe8eca9103c3d111564e6c in 1316ms, sequenceid=177, compaction requested=true 2023-04-15 02:54:12,282 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for b7b06ed62cbe8eca9103c3d111564e6c: 2023-04-15 02:54:12,282 DEBUG [RS:0;jenkins-hbase20:34681-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-15 02:54:12,282 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-15 02:54:12,282 DEBUG [RS:0;jenkins-hbase20:34681-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-15 02:54:12,282 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-15 02:54:12,282 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-15 02:54:12,283 DEBUG [RS:0;jenkins-hbase20:34681-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 84952 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-15 02:54:12,283 DEBUG [RS:0;jenkins-hbase20:34681-shortCompactions-0] regionserver.HStore(1912): b7b06ed62cbe8eca9103c3d111564e6c/A is initiating minor compaction (all files) 2023-04-15 02:54:12,283 INFO [RS:0;jenkins-hbase20:34681-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of b7b06ed62cbe8eca9103c3d111564e6c/A in TestAcidGuarantees,,1681527240182.b7b06ed62cbe8eca9103c3d111564e6c. 2023-04-15 02:54:12,283 INFO [RS:0;jenkins-hbase20:34681-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/b7b06ed62cbe8eca9103c3d111564e6c/A/a9fb2169d5184cbe8e4682c53928ba72, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/b7b06ed62cbe8eca9103c3d111564e6c/A/4f734e0147ea4a368d6ec7845146f73b, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/b7b06ed62cbe8eca9103c3d111564e6c/A/f062d2172d864cc7aac2415982fffc57] into tmpdir=hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/b7b06ed62cbe8eca9103c3d111564e6c/.tmp, totalSize=83.0 K 2023-04-15 02:54:12,283 DEBUG [RS:0;jenkins-hbase20:34681-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36382 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-15 02:54:12,283 DEBUG [RS:0;jenkins-hbase20:34681-longCompactions-0] regionserver.HStore(1912): b7b06ed62cbe8eca9103c3d111564e6c/B is initiating minor compaction (all files) 2023-04-15 02:54:12,283 INFO [RS:0;jenkins-hbase20:34681-longCompactions-0] regionserver.HRegion(2259): Starting compaction of b7b06ed62cbe8eca9103c3d111564e6c/B in TestAcidGuarantees,,1681527240182.b7b06ed62cbe8eca9103c3d111564e6c. 2023-04-15 02:54:12,283 INFO [RS:0;jenkins-hbase20:34681-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/b7b06ed62cbe8eca9103c3d111564e6c/B/b72caa2df2b34bf69d99434a82ee9ac0, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/b7b06ed62cbe8eca9103c3d111564e6c/B/11454a78c616407d849c8a14e7d1674a, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/b7b06ed62cbe8eca9103c3d111564e6c/B/9b76848f28af4f69ae6328d74ae603f8] into tmpdir=hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/b7b06ed62cbe8eca9103c3d111564e6c/.tmp, totalSize=35.5 K 2023-04-15 02:54:12,284 DEBUG [RS:0;jenkins-hbase20:34681-shortCompactions-0] compactions.Compactor(207): Compacting a9fb2169d5184cbe8e4682c53928ba72, keycount=150, bloomtype=ROW, size=25.6 K, encoding=NONE, compression=NONE, seqNum=144, earliestPutTs=1681527248703 2023-04-15 02:54:12,284 DEBUG [RS:0;jenkins-hbase20:34681-longCompactions-0] compactions.Compactor(207): Compacting b72caa2df2b34bf69d99434a82ee9ac0, keycount=150, bloomtype=ROW, size=12.0 K, encoding=NONE, compression=NONE, seqNum=144, earliestPutTs=1681527248703 2023-04-15 02:54:12,284 DEBUG [RS:0;jenkins-hbase20:34681-shortCompactions-0] compactions.Compactor(207): Compacting 4f734e0147ea4a368d6ec7845146f73b, keycount=150, bloomtype=ROW, size=25.3 K, encoding=NONE, compression=NONE, seqNum=163, earliestPutTs=1681527249855 2023-04-15 02:54:12,284 DEBUG [RS:0;jenkins-hbase20:34681-longCompactions-0] compactions.Compactor(207): Compacting 11454a78c616407d849c8a14e7d1674a, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=163, earliestPutTs=1681527249855 2023-04-15 02:54:12,284 DEBUG [RS:0;jenkins-hbase20:34681-shortCompactions-0] compactions.Compactor(207): Compacting f062d2172d864cc7aac2415982fffc57, keycount=200, bloomtype=ROW, size=32.1 K, encoding=NONE, compression=NONE, seqNum=177, earliestPutTs=1681527250839 2023-04-15 02:54:12,285 DEBUG [RS:0;jenkins-hbase20:34681-longCompactions-0] compactions.Compactor(207): Compacting 9b76848f28af4f69ae6328d74ae603f8, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=177, earliestPutTs=1681527250839 2023-04-15 02:54:12,295 INFO [RS:0;jenkins-hbase20:34681-longCompactions-0] throttle.PressureAwareThroughputController(145): b7b06ed62cbe8eca9103c3d111564e6c#B#compaction#136 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-15 02:54:12,307 INFO [RS:0;jenkins-hbase20:34681-shortCompactions-0] throttle.PressureAwareThroughputController(145): b7b06ed62cbe8eca9103c3d111564e6c#A#compaction#135 average throughput is 1.31 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-15 02:54:12,319 DEBUG [RS:0;jenkins-hbase20:34681-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/b7b06ed62cbe8eca9103c3d111564e6c/.tmp/B/e63856d5116b41328e79561838e6068c as hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/b7b06ed62cbe8eca9103c3d111564e6c/B/e63856d5116b41328e79561838e6068c 2023-04-15 02:54:12,324 INFO [RS:0;jenkins-hbase20:34681-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in b7b06ed62cbe8eca9103c3d111564e6c/B of b7b06ed62cbe8eca9103c3d111564e6c into e63856d5116b41328e79561838e6068c(size=12.1 K), total size for store is 12.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-15 02:54:12,324 DEBUG [RS:0;jenkins-hbase20:34681-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for b7b06ed62cbe8eca9103c3d111564e6c: 2023-04-15 02:54:12,324 INFO [RS:0;jenkins-hbase20:34681-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681527240182.b7b06ed62cbe8eca9103c3d111564e6c., storeName=b7b06ed62cbe8eca9103c3d111564e6c/B, priority=13, startTime=1681527252282; duration=0sec 2023-04-15 02:54:12,324 DEBUG [RS:0;jenkins-hbase20:34681-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-15 02:54:12,324 DEBUG [RS:0;jenkins-hbase20:34681-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 2 store files, 0 compacting, 2 eligible, 16 blocking 2023-04-15 02:54:12,325 DEBUG [RS:0;jenkins-hbase20:34681-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 0 files of size 0 starting at candidate #-1 after considering 0 permutations with 0 in ratio 2023-04-15 02:54:12,325 DEBUG [RS:0;jenkins-hbase20:34681-longCompactions-0] compactions.SortedCompactionPolicy(232): Not compacting files because we only have 0 files ready for compaction. Need 3 to initiate. 2023-04-15 02:54:12,325 DEBUG [RS:0;jenkins-hbase20:34681-longCompactions-0] regionserver.CompactSplit(415): Not compacting TestAcidGuarantees,,1681527240182.b7b06ed62cbe8eca9103c3d111564e6c. because compaction request was cancelled 2023-04-15 02:54:12,381 DEBUG [Thread-864] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-15 02:54:12,381 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-15 02:54:12,383 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-15 02:54:12,384 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-15 02:54:12,384 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-15 02:54:12,384 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-15 02:54:12,384 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-15 02:54:12,384 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-15 02:54:12,384 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-15 02:54:12,390 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:12,390 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:54:12,391 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:12,391 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-15 02:54:12,391 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-15 02:54:12,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-15 02:54:12,391 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:12,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:54:12,391 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-15 02:54:12,392 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:12,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-15 02:54:12,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:54:12,392 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-15 02:54:12,392 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-15 02:54:12,392 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-15 02:54:12,392 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-15 02:54:12,392 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-15 02:54:12,392 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-15 02:54:12,392 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-15 02:54:12,393 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-15 02:54:12,393 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681527240182.b7b06ed62cbe8eca9103c3d111564e6c. 2023-04-15 02:54:12,393 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681527240182.b7b06ed62cbe8eca9103c3d111564e6c. started... 2023-04-15 02:54:12,393 INFO [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing b7b06ed62cbe8eca9103c3d111564e6c 3/3 column families, dataSize=100.63 KB heapSize=299.58 KB 2023-04-15 02:54:12,393 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK b7b06ed62cbe8eca9103c3d111564e6c, store=A 2023-04-15 02:54:12,393 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-15 02:54:12,393 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK b7b06ed62cbe8eca9103c3d111564e6c, store=B 2023-04-15 02:54:12,393 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-15 02:54:12,393 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK b7b06ed62cbe8eca9103c3d111564e6c, store=C 2023-04-15 02:54:12,393 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-15 02:54:12,416 INFO [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230415df42d3450bf14f5097ffe9fc14ba834d to hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230415df42d3450bf14f5097ffe9fc14ba834d 2023-04-15 02:54:12,441 INFO [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=199, memsize=33.5 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/b7b06ed62cbe8eca9103c3d111564e6c/.tmp/A/6bd682223d1542898d764194cf4d3380 2023-04-15 02:54:12,852 INFO [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=199 (bloomFilter=true), to=hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/b7b06ed62cbe8eca9103c3d111564e6c/.tmp/B/7e797ffd65c5478e80443ace7c715fb7 2023-04-15 02:54:12,862 INFO [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=199 (bloomFilter=true), to=hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/b7b06ed62cbe8eca9103c3d111564e6c/.tmp/C/cd3f2f0d61b740ebbce4c8787269d3b1 2023-04-15 02:54:12,866 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/b7b06ed62cbe8eca9103c3d111564e6c/.tmp/A/6bd682223d1542898d764194cf4d3380 as hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/b7b06ed62cbe8eca9103c3d111564e6c/A/6bd682223d1542898d764194cf4d3380 2023-04-15 02:54:12,869 INFO [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/b7b06ed62cbe8eca9103c3d111564e6c/A/6bd682223d1542898d764194cf4d3380, entries=150, sequenceid=199, filesize=25.3 K 2023-04-15 02:54:12,874 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/b7b06ed62cbe8eca9103c3d111564e6c/.tmp/B/7e797ffd65c5478e80443ace7c715fb7 as hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/b7b06ed62cbe8eca9103c3d111564e6c/B/7e797ffd65c5478e80443ace7c715fb7 2023-04-15 02:54:12,878 INFO [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/b7b06ed62cbe8eca9103c3d111564e6c/B/7e797ffd65c5478e80443ace7c715fb7, entries=150, sequenceid=199, filesize=11.8 K 2023-04-15 02:54:12,879 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/b7b06ed62cbe8eca9103c3d111564e6c/.tmp/C/cd3f2f0d61b740ebbce4c8787269d3b1 as hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/b7b06ed62cbe8eca9103c3d111564e6c/C/cd3f2f0d61b740ebbce4c8787269d3b1 2023-04-15 02:54:12,895 INFO [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/b7b06ed62cbe8eca9103c3d111564e6c/C/cd3f2f0d61b740ebbce4c8787269d3b1, entries=150, sequenceid=199, filesize=11.8 K 2023-04-15 02:54:12,897 INFO [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~100.63 KB/103050, heapSize ~299.53 KB/306720, currentSize=0 B/0 for b7b06ed62cbe8eca9103c3d111564e6c in 504ms, sequenceid=199, compaction requested=true 2023-04-15 02:54:12,897 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for b7b06ed62cbe8eca9103c3d111564e6c: 2023-04-15 02:54:12,897 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681527240182.b7b06ed62cbe8eca9103c3d111564e6c. 2023-04-15 02:54:12,897 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-15 02:54:12,897 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-15 02:54:12,898 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:12,898 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-15 02:54:12,898 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34681,1681527209133' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-15 02:54:12,899 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:12,899 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:12,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:12,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:54:12,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:54:12,899 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:12,899 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-15 02:54:12,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:54:12,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:54:12,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:54:12,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:12,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:54:12,900 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34681,1681527209133' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-15 02:54:12,900 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2c290caf[Count = 0] remaining members to acquire global barrier 2023-04-15 02:54:12,900 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-15 02:54:12,900 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:12,901 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:12,901 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:12,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:12,901 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-15 02:54:12,901 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-15 02:54:12,901 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34681,1681527209133' in zk 2023-04-15 02:54:12,902 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:12,902 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-15 02:54:12,902 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-15 02:54:12,902 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-15 02:54:12,902 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-15 02:54:12,903 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:12,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:12,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:54:12,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:54:12,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:54:12,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:54:12,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:54:12,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:12,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:54:12,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:54:12,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:12,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34681,1681527209133': 2023-04-15 02:54:12,906 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34681,1681527209133' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-15 02:54:12,906 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-15 02:54:12,906 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-15 02:54:12,906 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-15 02:54:12,906 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-15 02:54:12,906 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-15 02:54:12,907 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:12,907 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-15 02:54:12,907 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:12,907 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:12,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:12,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:12,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:54:12,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:54:12,908 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-15 02:54:12,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-15 02:54:12,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:12,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:54:12,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:54:12,909 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:12,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:54:12,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:54:12,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:12,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:54:12,910 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:12,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:54:12,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:12,913 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-15 02:54:12,913 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:12,913 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-15 02:54:12,913 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:54:12,913 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-15 02:54:12,913 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:54:12,913 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-15 02:54:12,913 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:12,913 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:12,913 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:12,913 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-15 02:54:12,913 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-15 02:54:12,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-15 02:54:12,913 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:12,914 DEBUG [Thread-864] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-15 02:54:12,914 DEBUG [Thread-864] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-15 02:54:12,914 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-15 02:54:12,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-15 02:54:13,014 DEBUG [Thread-864] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-15 02:54:13,014 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-15 02:54:13,015 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-15 02:54:13,016 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-15 02:54:13,016 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-15 02:54:13,016 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-15 02:54:13,016 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-15 02:54:13,016 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-15 02:54:13,016 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-15 02:54:13,017 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:13,017 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:54:13,017 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:13,018 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:13,018 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-15 02:54:13,021 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-15 02:54:13,021 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-15 02:54:13,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-15 02:54:13,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:54:13,022 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:13,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-15 02:54:13,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:54:13,023 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-15 02:54:13,023 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-15 02:54:13,023 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-15 02:54:13,024 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-15 02:54:13,024 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-15 02:54:13,024 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-15 02:54:13,024 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-15 02:54:13,024 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-15 02:54:13,024 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681527240182.b7b06ed62cbe8eca9103c3d111564e6c. 2023-04-15 02:54:13,024 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681527240182.b7b06ed62cbe8eca9103c3d111564e6c. started... 2023-04-15 02:54:13,024 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for b7b06ed62cbe8eca9103c3d111564e6c: 2023-04-15 02:54:13,024 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681527240182.b7b06ed62cbe8eca9103c3d111564e6c. 2023-04-15 02:54:13,024 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-15 02:54:13,024 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-15 02:54:13,024 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:13,024 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-15 02:54:13,024 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34681,1681527209133' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-15 02:54:13,037 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:13,037 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:13,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:13,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:54:13,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:54:13,039 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:13,039 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-15 02:54:13,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:54:13,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:54:13,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:54:13,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:13,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:54:13,040 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34681,1681527209133' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-15 02:54:13,040 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@651f15f2[Count = 0] remaining members to acquire global barrier 2023-04-15 02:54:13,040 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-15 02:54:13,040 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:13,042 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:13,042 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:13,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:13,042 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-15 02:54:13,042 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-15 02:54:13,042 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34681,1681527209133' in zk 2023-04-15 02:54:13,042 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:13,042 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-15 02:54:13,043 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:13,043 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-15 02:54:13,043 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-15 02:54:13,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:13,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:54:13,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:54:13,043 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-15 02:54:13,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:54:13,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:54:13,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:54:13,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:13,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:54:13,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:54:13,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:13,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34681,1681527209133': 2023-04-15 02:54:13,045 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34681,1681527209133' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-15 02:54:13,045 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-15 02:54:13,045 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-15 02:54:13,045 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-15 02:54:13,045 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-15 02:54:13,045 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-15 02:54:13,046 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:13,046 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-15 02:54:13,046 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:13,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:13,047 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:13,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:13,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:54:13,047 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-15 02:54:13,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-15 02:54:13,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:54:13,047 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:13,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:13,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:54:13,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:54:13,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:54:13,048 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:13,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:54:13,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:13,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:54:13,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:54:13,050 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:13,050 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:54:13,050 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:54:13,050 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:13,050 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:13,050 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:13,050 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:13,050 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-15 02:54:13,050 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-15 02:54:13,050 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-15 02:54:13,050 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-15 02:54:13,050 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-15 02:54:13,050 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-15 02:54:13,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-15 02:54:13,050 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-15 02:54:13,050 DEBUG [Thread-864] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-15 02:54:13,051 DEBUG [Thread-864] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-15 02:54:13,051 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-15 02:54:13,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-15 02:54:13,135 DEBUG [RS:0;jenkins-hbase20:34681-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/b7b06ed62cbe8eca9103c3d111564e6c/.tmp/A/f789c3b8ecdb49d6ac6ba9ca495a5b11 as hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/b7b06ed62cbe8eca9103c3d111564e6c/A/f789c3b8ecdb49d6ac6ba9ca495a5b11 2023-04-15 02:54:13,141 INFO [RS:0;jenkins-hbase20:34681-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in b7b06ed62cbe8eca9103c3d111564e6c/A of b7b06ed62cbe8eca9103c3d111564e6c into f789c3b8ecdb49d6ac6ba9ca495a5b11(size=25.7 K), total size for store is 50.9 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-15 02:54:13,141 DEBUG [RS:0;jenkins-hbase20:34681-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for b7b06ed62cbe8eca9103c3d111564e6c: 2023-04-15 02:54:13,142 INFO [RS:0;jenkins-hbase20:34681-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681527240182.b7b06ed62cbe8eca9103c3d111564e6c., storeName=b7b06ed62cbe8eca9103c3d111564e6c/A, priority=13, startTime=1681527252282; duration=0sec 2023-04-15 02:54:13,142 DEBUG [RS:0;jenkins-hbase20:34681-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-15 02:54:13,151 DEBUG [Thread-864] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-15 02:54:13,151 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-15 02:54:13,153 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-15 02:54:13,154 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-15 02:54:13,154 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-15 02:54:13,154 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-15 02:54:13,154 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-15 02:54:13,154 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-15 02:54:13,154 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-15 02:54:13,154 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:13,154 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:54:13,155 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-15 02:54:13,155 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:13,155 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-15 02:54:13,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-15 02:54:13,155 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:13,155 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-15 02:54:13,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:54:13,156 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:13,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-15 02:54:13,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:54:13,156 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-15 02:54:13,170 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-15 02:54:13,170 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-15 02:54:13,170 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-15 02:54:13,170 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-15 02:54:13,170 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-15 02:54:13,171 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-15 02:54:13,171 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-15 02:54:13,171 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681527240182.b7b06ed62cbe8eca9103c3d111564e6c. 2023-04-15 02:54:13,171 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681527240182.b7b06ed62cbe8eca9103c3d111564e6c. started... 2023-04-15 02:54:13,171 INFO [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing b7b06ed62cbe8eca9103c3d111564e6c 3/3 column families, dataSize=13.42 KB heapSize=40.59 KB 2023-04-15 02:54:13,173 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK b7b06ed62cbe8eca9103c3d111564e6c, store=A 2023-04-15 02:54:13,173 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-15 02:54:13,173 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK b7b06ed62cbe8eca9103c3d111564e6c, store=B 2023-04-15 02:54:13,174 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-15 02:54:13,174 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK b7b06ed62cbe8eca9103c3d111564e6c, store=C 2023-04-15 02:54:13,174 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-15 02:54:13,202 INFO [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023041545c8ae0cb6d54fdfa28db895d93771e4 to hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023041545c8ae0cb6d54fdfa28db895d93771e4 2023-04-15 02:54:13,207 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34681] regionserver.HRegion(9158): Flush requested on b7b06ed62cbe8eca9103c3d111564e6c 2023-04-15 02:54:13,207 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681527240182.b7b06ed62cbe8eca9103c3d111564e6c. as already flushing 2023-04-15 02:54:13,227 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1681527253209, max timestamp=1681527253225Num uniques 859; 2023-04-15 02:54:13,227 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-8] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-15 02:54:13,227 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9564, totalHeapSize=21976, min timestamp=1681527253209, max timestamp=1681527253225Num uniques 859; 2023-04-15 02:54:13,227 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1681527253209, max timestamp=1681527253225Num uniques 859; 2023-04-15 02:54:13,227 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-3] regionserver.CompactionPipeline(163): Suffix data size=39334, new segment data size=9564, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-15 02:54:13,227 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-6] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-15 02:54:13,250 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=168, cellSize=7688, totalHeapSize=17712, min timestamp=1681527253242, max timestamp=1681527253248Num uniques 1068; 2023-04-15 02:54:13,250 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-5] regionserver.CompactionPipeline(163): Suffix data size=48908, new segment data size=7688, suffix heap size=139040,new segment heap size=17712 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1068, new segment cells count=168 2023-04-15 02:54:13,250 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=168, cellSize=7688, totalHeapSize=17712, min timestamp=1681527253242, max timestamp=1681527253248Num uniques 1068; 2023-04-15 02:54:13,250 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=168, cellSize=7688, totalHeapSize=17712, min timestamp=1681527253242, max timestamp=1681527253248Num uniques 1068; 2023-04-15 02:54:13,250 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-2] regionserver.CompactionPipeline(163): Suffix data size=48908, new segment data size=7688, suffix heap size=139040,new segment heap size=17712 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1068, new segment cells count=168 2023-04-15 02:54:13,250 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(163): Suffix data size=48908, new segment data size=7688, suffix heap size=139040,new segment heap size=17712 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1068, new segment cells count=168 2023-04-15 02:54:13,279 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=327, cellSize=14972, totalHeapSize=34248, min timestamp=1681527253268, max timestamp=1681527253277Num uniques 1027; 2023-04-15 02:54:13,279 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-1] regionserver.CompactionPipeline(163): Suffix data size=47032, new segment data size=14972, suffix heap size=134776,new segment heap size=34248 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1027, new segment cells count=327 2023-04-15 02:54:13,279 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=327, cellSize=14972, totalHeapSize=34248, min timestamp=1681527253268, max timestamp=1681527253277Num uniques 1027; 2023-04-15 02:54:13,280 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-7] regionserver.CompactionPipeline(163): Suffix data size=47032, new segment data size=14972, suffix heap size=134776,new segment heap size=34248 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1027, new segment cells count=327 2023-04-15 02:54:13,280 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=327, cellSize=14972, totalHeapSize=34248, min timestamp=1681527253268, max timestamp=1681527253277Num uniques 1027; 2023-04-15 02:54:13,280 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-4] regionserver.CompactionPipeline(163): Suffix data size=47032, new segment data size=14972, suffix heap size=134776,new segment heap size=34248 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1027, new segment cells count=327 2023-04-15 02:54:13,317 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=236, cellSize=10806, totalHeapSize=24784, min timestamp=1681527253293, max timestamp=1681527253304Num uniques 1186; 2023-04-15 02:54:13,317 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-9] regionserver.CompactionPipeline(163): Suffix data size=54316, new segment data size=10806, suffix heap size=151312,new segment heap size=24784 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1186, new segment cells count=236 2023-04-15 02:54:13,317 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=236, cellSize=10806, totalHeapSize=24784, min timestamp=1681527253293, max timestamp=1681527253304Num uniques 1186; 2023-04-15 02:54:13,318 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-8] regionserver.CompactionPipeline(163): Suffix data size=54316, new segment data size=10806, suffix heap size=151312,new segment heap size=24784 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1186, new segment cells count=236 2023-04-15 02:54:13,318 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=236, cellSize=10806, totalHeapSize=24784, min timestamp=1681527253293, max timestamp=1681527253304Num uniques 1186; 2023-04-15 02:54:13,318 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-3] regionserver.CompactionPipeline(163): Suffix data size=54316, new segment data size=10806, suffix heap size=151312,new segment heap size=24784 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1186, new segment cells count=236 2023-04-15 02:54:13,347 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=345, cellSize=15800, totalHeapSize=36120, min timestamp=1681527253333, max timestamp=1681527253345Num uniques 1095; 2023-04-15 02:54:13,348 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-5] regionserver.CompactionPipeline(163): Suffix data size=50150, new segment data size=15800, suffix heap size=141848,new segment heap size=36120 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1095, new segment cells count=345 2023-04-15 02:54:13,348 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=345, cellSize=15800, totalHeapSize=36120, min timestamp=1681527253333, max timestamp=1681527253345Num uniques 1095; 2023-04-15 02:54:13,348 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-6] regionserver.CompactionPipeline(163): Suffix data size=50150, new segment data size=15800, suffix heap size=141848,new segment heap size=36120 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1095, new segment cells count=345 2023-04-15 02:54:13,349 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=345, cellSize=15800, totalHeapSize=36120, min timestamp=1681527253333, max timestamp=1681527253345Num uniques 1095; 2023-04-15 02:54:13,349 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-2] regionserver.CompactionPipeline(163): Suffix data size=50150, new segment data size=15800, suffix heap size=141848,new segment heap size=36120 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1095, new segment cells count=345 2023-04-15 02:54:13,379 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=254, cellSize=11630, totalHeapSize=26656, min timestamp=1681527253365, max timestamp=1681527253377Num uniques 1204; 2023-04-15 02:54:13,379 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=254, cellSize=11630, totalHeapSize=26656, min timestamp=1681527253365, max timestamp=1681527253377Num uniques 1204; 2023-04-15 02:54:13,379 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(163): Suffix data size=55140, new segment data size=11630, suffix heap size=153184,new segment heap size=26656 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1204, new segment cells count=254 2023-04-15 02:54:13,379 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-1] regionserver.CompactionPipeline(163): Suffix data size=55140, new segment data size=11630, suffix heap size=153184,new segment heap size=26656 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1204, new segment cells count=254 2023-04-15 02:54:13,380 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=254, cellSize=11630, totalHeapSize=26656, min timestamp=1681527253365, max timestamp=1681527253377Num uniques 1204; 2023-04-15 02:54:13,380 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-7] regionserver.CompactionPipeline(163): Suffix data size=55140, new segment data size=11630, suffix heap size=153184,new segment heap size=26656 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1204, new segment cells count=254 2023-04-15 02:54:13,404 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=163, cellSize=7458, totalHeapSize=17192, min timestamp=1681527253395, max timestamp=1681527253402Num uniques 1113; 2023-04-15 02:54:13,404 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-9] regionserver.CompactionPipeline(163): Suffix data size=50968, new segment data size=7458, suffix heap size=143720,new segment heap size=17192 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1113, new segment cells count=163 2023-04-15 02:54:13,405 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=163, cellSize=7458, totalHeapSize=17192, min timestamp=1681527253395, max timestamp=1681527253402Num uniques 1113; 2023-04-15 02:54:13,405 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-4] regionserver.CompactionPipeline(163): Suffix data size=50968, new segment data size=7458, suffix heap size=143720,new segment heap size=17192 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1113, new segment cells count=163 2023-04-15 02:54:13,405 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=163, cellSize=7458, totalHeapSize=17192, min timestamp=1681527253395, max timestamp=1681527253402Num uniques 1113; 2023-04-15 02:54:13,405 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-8] regionserver.CompactionPipeline(163): Suffix data size=50968, new segment data size=7458, suffix heap size=143720,new segment heap size=17192 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1113, new segment cells count=163 2023-04-15 02:54:13,454 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=322, cellSize=14742, totalHeapSize=33728, min timestamp=1681527253434, max timestamp=1681527253451Num uniques 1022; 2023-04-15 02:54:13,454 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-3] regionserver.CompactionPipeline(163): Suffix data size=46802, new segment data size=14742, suffix heap size=134256,new segment heap size=33728 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1022, new segment cells count=322 2023-04-15 02:54:13,454 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=322, cellSize=14742, totalHeapSize=33728, min timestamp=1681527253434, max timestamp=1681527253451Num uniques 1022; 2023-04-15 02:54:13,454 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=322, cellSize=14742, totalHeapSize=33728, min timestamp=1681527253434, max timestamp=1681527253451Num uniques 1022; 2023-04-15 02:54:13,454 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-6] regionserver.CompactionPipeline(163): Suffix data size=46802, new segment data size=14742, suffix heap size=134256,new segment heap size=33728 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1022, new segment cells count=322 2023-04-15 02:54:13,454 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-5] regionserver.CompactionPipeline(163): Suffix data size=46802, new segment data size=14742, suffix heap size=134256,new segment heap size=33728 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1022, new segment cells count=322 2023-04-15 02:54:13,477 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=331, cellSize=15156, totalHeapSize=34664, min timestamp=1681527253467, max timestamp=1681527253474Num uniques 1181; 2023-04-15 02:54:13,477 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(163): Suffix data size=54086, new segment data size=15156, suffix heap size=150792,new segment heap size=34664 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1181, new segment cells count=331 2023-04-15 02:54:13,477 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=331, cellSize=15156, totalHeapSize=34664, min timestamp=1681527253467, max timestamp=1681527253474Num uniques 1181; 2023-04-15 02:54:13,478 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-2] regionserver.CompactionPipeline(163): Suffix data size=54086, new segment data size=15156, suffix heap size=150792,new segment heap size=34664 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1181, new segment cells count=331 2023-04-15 02:54:13,478 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=331, cellSize=15156, totalHeapSize=34664, min timestamp=1681527253467, max timestamp=1681527253474Num uniques 1181; 2023-04-15 02:54:13,478 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-1] regionserver.CompactionPipeline(163): Suffix data size=54086, new segment data size=15156, suffix heap size=150792,new segment heap size=34664 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1181, new segment cells count=331 2023-04-15 02:54:13,512 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=290, cellSize=13280, totalHeapSize=30400, min timestamp=1681527253496, max timestamp=1681527253510Num uniques 1190; 2023-04-15 02:54:13,512 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-9] regionserver.CompactionPipeline(163): Suffix data size=54500, new segment data size=13280, suffix heap size=151728,new segment heap size=30400 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1190, new segment cells count=290 2023-04-15 02:54:13,512 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=290, cellSize=13280, totalHeapSize=30400, min timestamp=1681527253496, max timestamp=1681527253510Num uniques 1190; 2023-04-15 02:54:13,512 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-7] regionserver.CompactionPipeline(163): Suffix data size=54500, new segment data size=13280, suffix heap size=151728,new segment heap size=30400 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1190, new segment cells count=290 2023-04-15 02:54:13,513 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=290, cellSize=13280, totalHeapSize=30400, min timestamp=1681527253496, max timestamp=1681527253510Num uniques 1190; 2023-04-15 02:54:13,513 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-4] regionserver.CompactionPipeline(163): Suffix data size=54500, new segment data size=13280, suffix heap size=151728,new segment heap size=30400 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1190, new segment cells count=290 2023-04-15 02:54:13,536 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=249, cellSize=11404, totalHeapSize=26136, min timestamp=1681527253524, max timestamp=1681527253534Num uniques 1149; 2023-04-15 02:54:13,536 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-6] regionserver.CompactionPipeline(163): Suffix data size=52624, new segment data size=11404, suffix heap size=147464,new segment heap size=26136 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1149, new segment cells count=249 2023-04-15 02:54:13,536 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=249, cellSize=11404, totalHeapSize=26136, min timestamp=1681527253524, max timestamp=1681527253534Num uniques 1149; 2023-04-15 02:54:13,536 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=249, cellSize=11404, totalHeapSize=26136, min timestamp=1681527253524, max timestamp=1681527253534Num uniques 1149; 2023-04-15 02:54:13,536 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-8] regionserver.CompactionPipeline(163): Suffix data size=52624, new segment data size=11404, suffix heap size=147464,new segment heap size=26136 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1149, new segment cells count=249 2023-04-15 02:54:13,536 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-3] regionserver.CompactionPipeline(163): Suffix data size=52624, new segment data size=11404, suffix heap size=147464,new segment heap size=26136 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1149, new segment cells count=249 2023-04-15 02:54:13,566 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=258, cellSize=11810, totalHeapSize=27072, min timestamp=1681527253554, max timestamp=1681527253564Num uniques 1108; 2023-04-15 02:54:13,566 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(163): Suffix data size=50740, new segment data size=11810, suffix heap size=143200,new segment heap size=27072 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1108, new segment cells count=258 2023-04-15 02:54:13,566 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=258, cellSize=11810, totalHeapSize=27072, min timestamp=1681527253554, max timestamp=1681527253564Num uniques 1108; 2023-04-15 02:54:13,566 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-5] regionserver.CompactionPipeline(163): Suffix data size=50740, new segment data size=11810, suffix heap size=143200,new segment heap size=27072 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1108, new segment cells count=258 2023-04-15 02:54:13,567 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=258, cellSize=11810, totalHeapSize=27072, min timestamp=1681527253554, max timestamp=1681527253564Num uniques 1108; 2023-04-15 02:54:13,567 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-2] regionserver.CompactionPipeline(163): Suffix data size=50740, new segment data size=11810, suffix heap size=143200,new segment heap size=27072 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1108, new segment cells count=258 2023-04-15 02:54:13,594 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=217, cellSize=9932, totalHeapSize=22808, min timestamp=1681527253579, max timestamp=1681527253592Num uniques 1117; 2023-04-15 02:54:13,594 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-1] regionserver.CompactionPipeline(163): Suffix data size=51152, new segment data size=9932, suffix heap size=144136,new segment heap size=22808 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1117, new segment cells count=217 2023-04-15 02:54:13,594 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=217, cellSize=9932, totalHeapSize=22808, min timestamp=1681527253579, max timestamp=1681527253592Num uniques 1117; 2023-04-15 02:54:13,594 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-9] regionserver.CompactionPipeline(163): Suffix data size=51152, new segment data size=9932, suffix heap size=144136,new segment heap size=22808 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1117, new segment cells count=217 2023-04-15 02:54:13,595 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=217, cellSize=9932, totalHeapSize=22808, min timestamp=1681527253579, max timestamp=1681527253592Num uniques 1117; 2023-04-15 02:54:13,595 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-1] regionserver.CompactionPipeline(163): Suffix data size=51152, new segment data size=9932, suffix heap size=144136,new segment heap size=22808 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1117, new segment cells count=217 2023-04-15 02:54:13,616 INFO [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=207, memsize=6.7 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/b7b06ed62cbe8eca9103c3d111564e6c/.tmp/A/acdbc16f0bd4470e853d39d9237b4be9 2023-04-15 02:54:13,623 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=276, cellSize=12636, totalHeapSize=28944, min timestamp=1681527253600, max timestamp=1681527253621Num uniques 1076; 2023-04-15 02:54:13,624 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-4] regionserver.CompactionPipeline(163): Suffix data size=49276, new segment data size=12636, suffix heap size=139872,new segment heap size=28944 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1076, new segment cells count=276 2023-04-15 02:54:13,625 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=276, cellSize=12636, totalHeapSize=28944, min timestamp=1681527253600, max timestamp=1681527253621Num uniques 1076; 2023-04-15 02:54:13,626 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-4] regionserver.CompactionPipeline(163): Suffix data size=49276, new segment data size=12636, suffix heap size=139872,new segment heap size=28944 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1076, new segment cells count=276 2023-04-15 02:54:13,627 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=276, cellSize=12636, totalHeapSize=28944, min timestamp=1681527253600, max timestamp=1681527253621Num uniques 1076; 2023-04-15 02:54:13,627 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-4] regionserver.CompactionPipeline(163): Suffix data size=49276, new segment data size=12636, suffix heap size=139872,new segment heap size=28944 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1076, new segment cells count=276 2023-04-15 02:54:13,651 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=235, cellSize=10760, totalHeapSize=24680, min timestamp=1681527253629, max timestamp=1681527253649Num uniques 1135; 2023-04-15 02:54:13,651 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-3] regionserver.CompactionPipeline(163): Suffix data size=51980, new segment data size=10760, suffix heap size=146008,new segment heap size=24680 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1135, new segment cells count=235 2023-04-15 02:54:13,651 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=235, cellSize=10760, totalHeapSize=24680, min timestamp=1681527253629, max timestamp=1681527253649Num uniques 1135; 2023-04-15 02:54:13,652 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(163): Suffix data size=51980, new segment data size=10760, suffix heap size=146008,new segment heap size=24680 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1135, new segment cells count=235 2023-04-15 02:54:13,652 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=235, cellSize=10760, totalHeapSize=24680, min timestamp=1681527253629, max timestamp=1681527253649Num uniques 1135; 2023-04-15 02:54:13,652 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-5] regionserver.CompactionPipeline(163): Suffix data size=51980, new segment data size=10760, suffix heap size=146008,new segment heap size=24680 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1135, new segment cells count=235 2023-04-15 02:54:13,678 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=194, cellSize=8884, totalHeapSize=20416, min timestamp=1681527253671, max timestamp=1681527253675Num uniques 1094; 2023-04-15 02:54:13,678 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=194, cellSize=8884, totalHeapSize=20416, min timestamp=1681527253671, max timestamp=1681527253675Num uniques 1094; 2023-04-15 02:54:13,678 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-7] regionserver.CompactionPipeline(163): Suffix data size=50104, new segment data size=8884, suffix heap size=141744,new segment heap size=20416 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1094, new segment cells count=194 2023-04-15 02:54:13,678 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-2] regionserver.CompactionPipeline(163): Suffix data size=50104, new segment data size=8884, suffix heap size=141744,new segment heap size=20416 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1094, new segment cells count=194 2023-04-15 02:54:13,680 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=194, cellSize=8884, totalHeapSize=20416, min timestamp=1681527253671, max timestamp=1681527253675Num uniques 1094; 2023-04-15 02:54:13,680 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-9] regionserver.CompactionPipeline(163): Suffix data size=50104, new segment data size=8884, suffix heap size=141744,new segment heap size=20416 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1094, new segment cells count=194 2023-04-15 02:54:13,709 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=303, cellSize=13875, totalHeapSize=31752, min timestamp=1681527253693, max timestamp=1681527253706Num uniques 1053; 2023-04-15 02:54:13,709 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=303, cellSize=13875, totalHeapSize=31752, min timestamp=1681527253693, max timestamp=1681527253706Num uniques 1053; 2023-04-15 02:54:13,709 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-1] regionserver.CompactionPipeline(163): Suffix data size=48225, new segment data size=13875, suffix heap size=137480,new segment heap size=31752 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1053, new segment cells count=303 2023-04-15 02:54:13,709 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=303, cellSize=13875, totalHeapSize=31752, min timestamp=1681527253693, max timestamp=1681527253706Num uniques 1053; 2023-04-15 02:54:13,709 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-4] regionserver.CompactionPipeline(163): Suffix data size=48225, new segment data size=13875, suffix heap size=137480,new segment heap size=31752 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1053, new segment cells count=303 2023-04-15 02:54:13,709 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-6] regionserver.CompactionPipeline(163): Suffix data size=48225, new segment data size=13875, suffix heap size=137480,new segment heap size=31752 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1053, new segment cells count=303 2023-04-15 02:54:13,738 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b7b06ed62cbe8eca9103c3d111564e6c, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:54:13,739 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=212, cellSize=9702, totalHeapSize=22288, min timestamp=1681527253727, max timestamp=1681527253736Num uniques 1162; 2023-04-15 02:54:13,739 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34681] ipc.CallRunner(144): callId: 289 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:39106 deadline: 1681527313736, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b7b06ed62cbe8eca9103c3d111564e6c, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:13,739 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-8] regionserver.CompactionPipeline(163): Suffix data size=53212, new segment data size=9702, suffix heap size=148816,new segment heap size=22288 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1162, new segment cells count=212 2023-04-15 02:54:13,739 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=212, cellSize=9702, totalHeapSize=22288, min timestamp=1681527253727, max timestamp=1681527253736Num uniques 1162; 2023-04-15 02:54:13,739 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=212, cellSize=9702, totalHeapSize=22288, min timestamp=1681527253727, max timestamp=1681527253736Num uniques 1162; 2023-04-15 02:54:13,739 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-3] regionserver.CompactionPipeline(163): Suffix data size=53212, new segment data size=9702, suffix heap size=148816,new segment heap size=22288 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1162, new segment cells count=212 2023-04-15 02:54:13,739 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(163): Suffix data size=53212, new segment data size=9702, suffix heap size=148816,new segment heap size=22288 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1162, new segment cells count=212 2023-04-15 02:54:13,779 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=221, cellSize=10116, totalHeapSize=23224, min timestamp=1681527253756, max timestamp=1681527253777Num uniques 1071; 2023-04-15 02:54:13,779 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=221, cellSize=10116, totalHeapSize=23224, min timestamp=1681527253756, max timestamp=1681527253777Num uniques 1071; 2023-04-15 02:54:13,779 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=221, cellSize=10116, totalHeapSize=23224, min timestamp=1681527253756, max timestamp=1681527253777Num uniques 1071; 2023-04-15 02:54:13,779 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-5] regionserver.CompactionPipeline(163): Suffix data size=49046, new segment data size=10116, suffix heap size=139352,new segment heap size=23224 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1071, new segment cells count=221 2023-04-15 02:54:13,779 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-2] regionserver.CompactionPipeline(163): Suffix data size=49046, new segment data size=10116, suffix heap size=139352,new segment heap size=23224 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1071, new segment cells count=221 2023-04-15 02:54:13,779 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-7] regionserver.CompactionPipeline(163): Suffix data size=49046, new segment data size=10116, suffix heap size=139352,new segment heap size=23224 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1071, new segment cells count=221 2023-04-15 02:54:13,829 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=330, cellSize=15110, totalHeapSize=34560, min timestamp=1681527253806, max timestamp=1681527253827Num uniques 1080; 2023-04-15 02:54:13,829 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-1] regionserver.CompactionPipeline(163): Suffix data size=49460, new segment data size=15110, suffix heap size=140288,new segment heap size=34560 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1080, new segment cells count=330 2023-04-15 02:54:13,829 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=330, cellSize=15110, totalHeapSize=34560, min timestamp=1681527253806, max timestamp=1681527253827Num uniques 1080; 2023-04-15 02:54:13,829 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-9] regionserver.CompactionPipeline(163): Suffix data size=49460, new segment data size=15110, suffix heap size=140288,new segment heap size=34560 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1080, new segment cells count=330 2023-04-15 02:54:13,830 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=330, cellSize=15110, totalHeapSize=34560, min timestamp=1681527253806, max timestamp=1681527253827Num uniques 1080; 2023-04-15 02:54:13,830 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-4] regionserver.CompactionPipeline(163): Suffix data size=49460, new segment data size=15110, suffix heap size=140288,new segment heap size=34560 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1080, new segment cells count=330 2023-04-15 02:54:13,860 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=339, cellSize=15524, totalHeapSize=35496, min timestamp=1681527253848, max timestamp=1681527253857Num uniques 1189; 2023-04-15 02:54:13,860 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-6] regionserver.CompactionPipeline(163): Suffix data size=54454, new segment data size=15524, suffix heap size=151624,new segment heap size=35496 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1189, new segment cells count=339 2023-04-15 02:54:13,860 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=339, cellSize=15524, totalHeapSize=35496, min timestamp=1681527253848, max timestamp=1681527253857Num uniques 1189; 2023-04-15 02:54:13,860 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=339, cellSize=15524, totalHeapSize=35496, min timestamp=1681527253848, max timestamp=1681527253857Num uniques 1189; 2023-04-15 02:54:13,860 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-3] regionserver.CompactionPipeline(163): Suffix data size=54454, new segment data size=15524, suffix heap size=151624,new segment heap size=35496 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1189, new segment cells count=339 2023-04-15 02:54:13,860 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-8] regionserver.CompactionPipeline(163): Suffix data size=54454, new segment data size=15524, suffix heap size=151624,new segment heap size=35496 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1189, new segment cells count=339 2023-04-15 02:54:13,887 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=348, cellSize=15938, totalHeapSize=36432, min timestamp=1681527253871, max timestamp=1681527253885Num uniques 1198; 2023-04-15 02:54:13,887 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(163): Suffix data size=54868, new segment data size=15938, suffix heap size=152560,new segment heap size=36432 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1198, new segment cells count=348 2023-04-15 02:54:13,889 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=348, cellSize=15938, totalHeapSize=36432, min timestamp=1681527253871, max timestamp=1681527253885Num uniques 1198; 2023-04-15 02:54:13,889 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-5] regionserver.CompactionPipeline(163): Suffix data size=54868, new segment data size=15938, suffix heap size=152560,new segment heap size=36432 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1198, new segment cells count=348 2023-04-15 02:54:13,889 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=348, cellSize=15938, totalHeapSize=36432, min timestamp=1681527253871, max timestamp=1681527253885Num uniques 1198; 2023-04-15 02:54:13,889 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-2] regionserver.CompactionPipeline(163): Suffix data size=54868, new segment data size=15938, suffix heap size=152560,new segment heap size=36432 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1198, new segment cells count=348 2023-04-15 02:54:13,934 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=157, cellSize=7185, totalHeapSize=16568, min timestamp=1681527253916, max timestamp=1681527253932Num uniques 1207; 2023-04-15 02:54:13,934 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=157, cellSize=7185, totalHeapSize=16568, min timestamp=1681527253916, max timestamp=1681527253932Num uniques 1207; 2023-04-15 02:54:13,934 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=157, cellSize=7185, totalHeapSize=16568, min timestamp=1681527253916, max timestamp=1681527253932Num uniques 1207; 2023-04-15 02:54:13,934 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-9] regionserver.CompactionPipeline(163): Suffix data size=55275, new segment data size=7185, suffix heap size=153496,new segment heap size=16568 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1207, new segment cells count=157 2023-04-15 02:54:13,934 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-1] regionserver.CompactionPipeline(163): Suffix data size=55275, new segment data size=7185, suffix heap size=153496,new segment heap size=16568 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1207, new segment cells count=157 2023-04-15 02:54:13,934 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-7] regionserver.CompactionPipeline(163): Suffix data size=55275, new segment data size=7185, suffix heap size=153496,new segment heap size=16568 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1207, new segment cells count=157 2023-04-15 02:54:13,967 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=266, cellSize=12176, totalHeapSize=27904, min timestamp=1681527253951, max timestamp=1681527253965Num uniques 1016; 2023-04-15 02:54:13,967 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-4] regionserver.CompactionPipeline(163): Suffix data size=46526, new segment data size=12176, suffix heap size=133632,new segment heap size=27904 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1016, new segment cells count=266 2023-04-15 02:54:13,968 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=266, cellSize=12176, totalHeapSize=27904, min timestamp=1681527253951, max timestamp=1681527253965Num uniques 1016; 2023-04-15 02:54:13,968 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=266, cellSize=12176, totalHeapSize=27904, min timestamp=1681527253951, max timestamp=1681527253965Num uniques 1016; 2023-04-15 02:54:13,968 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-6] regionserver.CompactionPipeline(163): Suffix data size=46526, new segment data size=12176, suffix heap size=133632,new segment heap size=27904 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1016, new segment cells count=266 2023-04-15 02:54:13,968 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-3] regionserver.CompactionPipeline(163): Suffix data size=46526, new segment data size=12176, suffix heap size=133632,new segment heap size=27904 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1016, new segment cells count=266 2023-04-15 02:54:14,004 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=225, cellSize=10300, totalHeapSize=23640, min timestamp=1681527253984, max timestamp=1681527254001Num uniques 1125; 2023-04-15 02:54:14,004 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(163): Suffix data size=51520, new segment data size=10300, suffix heap size=144968,new segment heap size=23640 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1125, new segment cells count=225 2023-04-15 02:54:14,004 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=225, cellSize=10300, totalHeapSize=23640, min timestamp=1681527253984, max timestamp=1681527254001Num uniques 1125; 2023-04-15 02:54:14,004 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-5] regionserver.CompactionPipeline(163): Suffix data size=51520, new segment data size=10300, suffix heap size=144968,new segment heap size=23640 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1125, new segment cells count=225 2023-04-15 02:54:14,004 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=375, cellSize=17170, totalHeapSize=39240, min timestamp=1681527253983, max timestamp=1681527254001Num uniques 1125; 2023-04-15 02:54:14,005 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-8] regionserver.CompactionPipeline(163): Suffix data size=51520, new segment data size=17170, suffix heap size=144968,new segment heap size=39240 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1125, new segment cells count=375 2023-04-15 02:54:14,026 INFO [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=6.71 KB at sequenceid=207 (bloomFilter=true), to=hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/b7b06ed62cbe8eca9103c3d111564e6c/.tmp/B/860fa6f1f1514f058db8f8453e5c1fc7 2023-04-15 02:54:14,045 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=334, cellSize=15294, totalHeapSize=34976, min timestamp=1681527254026, max timestamp=1681527254042Num uniques 1084; 2023-04-15 02:54:14,045 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-1] regionserver.CompactionPipeline(163): Suffix data size=49644, new segment data size=15294, suffix heap size=140704,new segment heap size=34976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1084, new segment cells count=334 2023-04-15 02:54:14,045 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=334, cellSize=15294, totalHeapSize=34976, min timestamp=1681527254026, max timestamp=1681527254042Num uniques 1234; 2023-04-15 02:54:14,045 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-2] regionserver.CompactionPipeline(163): Suffix data size=56514, new segment data size=15294, suffix heap size=156304,new segment heap size=34976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1234, new segment cells count=334 2023-04-15 02:54:14,045 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=334, cellSize=15294, totalHeapSize=34976, min timestamp=1681527254026, max timestamp=1681527254042Num uniques 1084; 2023-04-15 02:54:14,045 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-9] regionserver.CompactionPipeline(163): Suffix data size=49644, new segment data size=15294, suffix heap size=140704,new segment heap size=34976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1084, new segment cells count=334 2023-04-15 02:54:14,047 INFO [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=6.71 KB at sequenceid=207 (bloomFilter=true), to=hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/b7b06ed62cbe8eca9103c3d111564e6c/.tmp/C/6630954ef78e4fa2a6911d94d6b1daba 2023-04-15 02:54:14,055 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/b7b06ed62cbe8eca9103c3d111564e6c/.tmp/A/acdbc16f0bd4470e853d39d9237b4be9 as hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/b7b06ed62cbe8eca9103c3d111564e6c/A/acdbc16f0bd4470e853d39d9237b4be9 2023-04-15 02:54:14,061 INFO [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/b7b06ed62cbe8eca9103c3d111564e6c/A/acdbc16f0bd4470e853d39d9237b4be9, entries=150, sequenceid=207, filesize=25.3 K 2023-04-15 02:54:14,063 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/b7b06ed62cbe8eca9103c3d111564e6c/.tmp/B/860fa6f1f1514f058db8f8453e5c1fc7 as hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/b7b06ed62cbe8eca9103c3d111564e6c/B/860fa6f1f1514f058db8f8453e5c1fc7 2023-04-15 02:54:14,068 INFO [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/b7b06ed62cbe8eca9103c3d111564e6c/B/860fa6f1f1514f058db8f8453e5c1fc7, entries=100, sequenceid=207, filesize=9.4 K 2023-04-15 02:54:14,069 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/b7b06ed62cbe8eca9103c3d111564e6c/.tmp/C/6630954ef78e4fa2a6911d94d6b1daba as hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/b7b06ed62cbe8eca9103c3d111564e6c/C/6630954ef78e4fa2a6911d94d6b1daba 2023-04-15 02:54:14,075 INFO [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/b7b06ed62cbe8eca9103c3d111564e6c/C/6630954ef78e4fa2a6911d94d6b1daba, entries=100, sequenceid=207, filesize=9.4 K 2023-04-15 02:54:14,079 INFO [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~20.13 KB/20610, heapSize ~60.47 KB/61920, currentSize=120.76 KB/123660 for b7b06ed62cbe8eca9103c3d111564e6c in 908ms, sequenceid=207, compaction requested=true 2023-04-15 02:54:14,079 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for b7b06ed62cbe8eca9103c3d111564e6c: 2023-04-15 02:54:14,079 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681527240182.b7b06ed62cbe8eca9103c3d111564e6c. 2023-04-15 02:54:14,079 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-15 02:54:14,079 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-15 02:54:14,079 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:14,079 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-15 02:54:14,079 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34681,1681527209133' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-15 02:54:14,079 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34681] regionserver.HRegion(9158): Flush requested on b7b06ed62cbe8eca9103c3d111564e6c 2023-04-15 02:54:14,080 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing b7b06ed62cbe8eca9103c3d111564e6c 3/3 column families, dataSize=127.47 KB heapSize=348.66 KB 2023-04-15 02:54:14,080 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK b7b06ed62cbe8eca9103c3d111564e6c, store=A 2023-04-15 02:54:14,080 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-15 02:54:14,080 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK b7b06ed62cbe8eca9103c3d111564e6c, store=B 2023-04-15 02:54:14,080 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-15 02:54:14,080 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK b7b06ed62cbe8eca9103c3d111564e6c, store=C 2023-04-15 02:54:14,080 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:14,080 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-15 02:54:14,080 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:14,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:14,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:54:14,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:54:14,080 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:14,080 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-15 02:54:14,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:54:14,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:54:14,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:54:14,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:14,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:54:14,082 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34681,1681527209133' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-15 02:54:14,082 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@130d22f8[Count = 0] remaining members to acquire global barrier 2023-04-15 02:54:14,082 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-15 02:54:14,082 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:14,083 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:14,083 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:14,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:14,083 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-15 02:54:14,083 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-15 02:54:14,083 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34681,1681527209133' in zk 2023-04-15 02:54:14,084 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:14,084 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-15 02:54:14,086 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-15 02:54:14,086 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-15 02:54:14,086 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:14,086 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-15 02:54:14,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:14,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:54:14,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:54:14,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:54:14,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:54:14,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:54:14,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:14,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:54:14,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:54:14,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:14,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34681,1681527209133': 2023-04-15 02:54:14,089 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34681,1681527209133' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-15 02:54:14,089 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-15 02:54:14,090 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-15 02:54:14,090 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-15 02:54:14,090 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-15 02:54:14,090 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-15 02:54:14,091 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:14,091 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-15 02:54:14,091 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:14,092 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:14,092 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:14,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:14,092 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:14,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:14,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:54:14,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:54:14,092 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-15 02:54:14,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-15 02:54:14,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:14,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:54:14,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:54:14,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:54:14,094 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-15 02:54:14,094 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-15 02:54:14,094 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-15 02:54:14,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-15 02:54:14,094 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:14,094 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:54:14,094 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-15 02:54:14,094 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-15 02:54:14,094 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-15 02:54:14,094 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:54:14,094 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:14,094 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:14,094 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:14,094 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:14,094 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-15 02:54:14,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-15 02:54:14,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:54:14,095 DEBUG [Thread-864] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-15 02:54:14,095 DEBUG [Thread-864] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-15 02:54:14,096 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-15 02:54:14,103 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b7b06ed62cbe8eca9103c3d111564e6c, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:54:14,103 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34681] ipc.CallRunner(144): callId: 378 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:39132 deadline: 1681527314100, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b7b06ed62cbe8eca9103c3d111564e6c, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:14,104 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b7b06ed62cbe8eca9103c3d111564e6c, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:54:14,104 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34681] ipc.CallRunner(144): callId: 351 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:39106 deadline: 1681527314101, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b7b06ed62cbe8eca9103c3d111564e6c, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:14,195 DEBUG [Thread-864] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-15 02:54:14,195 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-15 02:54:14,196 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-15 02:54:14,197 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-15 02:54:14,197 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-15 02:54:14,197 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-15 02:54:14,197 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-15 02:54:14,198 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-15 02:54:14,198 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-15 02:54:14,198 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:14,198 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:54:14,198 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:14,199 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-15 02:54:14,199 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-15 02:54:14,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-15 02:54:14,199 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:14,199 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-15 02:54:14,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:54:14,199 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:14,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-15 02:54:14,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:54:14,199 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-15 02:54:14,200 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-15 02:54:14,200 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-15 02:54:14,200 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-15 02:54:14,200 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-15 02:54:14,200 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-15 02:54:14,200 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-15 02:54:14,200 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-15 02:54:14,200 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681527240182.b7b06ed62cbe8eca9103c3d111564e6c. 2023-04-15 02:54:14,200 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681527240182.b7b06ed62cbe8eca9103c3d111564e6c. started... 2023-04-15 02:54:14,200 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681527240182.b7b06ed62cbe8eca9103c3d111564e6c. as already flushing 2023-04-15 02:54:14,200 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681527240182.b7b06ed62cbe8eca9103c3d111564e6c. 2023-04-15 02:54:14,200 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-15 02:54:14,200 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-15 02:54:14,200 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:14,200 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-15 02:54:14,200 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34681,1681527209133' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-15 02:54:14,201 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:14,201 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:14,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:14,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:54:14,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:54:14,201 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:14,202 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-15 02:54:14,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:54:14,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:54:14,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:54:14,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:14,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:54:14,203 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34681,1681527209133' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-15 02:54:14,203 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3ce77096[Count = 0] remaining members to acquire global barrier 2023-04-15 02:54:14,203 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-15 02:54:14,203 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:14,203 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:14,203 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:14,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:14,203 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:14,203 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-15 02:54:14,203 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-15 02:54:14,203 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-15 02:54:14,203 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34681,1681527209133' in zk 2023-04-15 02:54:14,204 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:14,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:14,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:54:14,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:54:14,204 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-15 02:54:14,204 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-15 02:54:14,204 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-15 02:54:14,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:54:14,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:54:14,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:54:14,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:14,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:54:14,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:54:14,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:14,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34681,1681527209133': 2023-04-15 02:54:14,206 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34681,1681527209133' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-15 02:54:14,206 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-15 02:54:14,206 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-15 02:54:14,206 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-15 02:54:14,206 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-15 02:54:14,206 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-15 02:54:14,207 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b7b06ed62cbe8eca9103c3d111564e6c, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:54:14,207 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:14,207 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34681] ipc.CallRunner(144): callId: 380 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:39132 deadline: 1681527314205, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b7b06ed62cbe8eca9103c3d111564e6c, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:14,207 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:14,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:14,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:54:14,207 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b7b06ed62cbe8eca9103c3d111564e6c, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:54:14,207 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-15 02:54:14,207 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:14,207 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34681] ipc.CallRunner(144): callId: 353 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:39106 deadline: 1681527314205, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b7b06ed62cbe8eca9103c3d111564e6c, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:14,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:54:14,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:14,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:54:14,208 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:14,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:54:14,208 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-15 02:54:14,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-15 02:54:14,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:54:14,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:14,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:54:14,209 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:14,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:14,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:54:14,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:54:14,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:14,211 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-15 02:54:14,211 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-15 02:54:14,211 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-15 02:54:14,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-15 02:54:14,211 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:14,211 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:54:14,211 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:54:14,211 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-15 02:54:14,211 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-15 02:54:14,211 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-15 02:54:14,211 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-15 02:54:14,211 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:14,211 DEBUG [Thread-864] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-15 02:54:14,211 DEBUG [Thread-864] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-15 02:54:14,211 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-15 02:54:14,211 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:14,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-15 02:54:14,212 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:14,212 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:14,311 DEBUG [Thread-864] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-15 02:54:14,312 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-15 02:54:14,313 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-15 02:54:14,314 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-15 02:54:14,314 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-15 02:54:14,314 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-15 02:54:14,314 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-15 02:54:14,314 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-15 02:54:14,314 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-15 02:54:14,314 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:14,314 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:54:14,315 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:14,315 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-15 02:54:14,315 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-15 02:54:14,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-15 02:54:14,315 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:14,315 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-15 02:54:14,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:54:14,316 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:14,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-15 02:54:14,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:54:14,316 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-15 02:54:14,316 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-15 02:54:14,316 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-15 02:54:14,316 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-15 02:54:14,316 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-15 02:54:14,316 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-15 02:54:14,316 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-15 02:54:14,316 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-15 02:54:14,316 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681527240182.b7b06ed62cbe8eca9103c3d111564e6c. 2023-04-15 02:54:14,316 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681527240182.b7b06ed62cbe8eca9103c3d111564e6c. started... 2023-04-15 02:54:14,317 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681527240182.b7b06ed62cbe8eca9103c3d111564e6c. as already flushing 2023-04-15 02:54:14,317 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681527240182.b7b06ed62cbe8eca9103c3d111564e6c. 2023-04-15 02:54:14,317 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-15 02:54:14,317 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-15 02:54:14,317 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:14,317 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-15 02:54:14,317 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34681,1681527209133' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-15 02:54:14,318 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:14,318 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:14,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:14,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:54:14,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:54:14,319 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:14,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:54:14,319 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-15 02:54:14,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:54:14,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:54:14,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:14,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:54:14,323 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34681,1681527209133' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-15 02:54:14,323 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@58cf4fd1[Count = 0] remaining members to acquire global barrier 2023-04-15 02:54:14,323 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-15 02:54:14,323 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:14,324 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:14,324 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:14,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:14,324 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:14,324 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-15 02:54:14,324 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-15 02:54:14,324 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-15 02:54:14,324 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34681,1681527209133' in zk 2023-04-15 02:54:14,325 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-15 02:54:14,325 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:14,325 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-15 02:54:14,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:14,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:54:14,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:54:14,325 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-15 02:54:14,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:54:14,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:54:14,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:54:14,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:14,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:54:14,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:54:14,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:14,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34681,1681527209133': 2023-04-15 02:54:14,327 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34681,1681527209133' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-15 02:54:14,327 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-15 02:54:14,327 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-15 02:54:14,327 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-15 02:54:14,327 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-15 02:54:14,327 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-15 02:54:14,328 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:14,328 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:14,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:14,328 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:14,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:54:14,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:54:14,328 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-15 02:54:14,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:14,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:54:14,328 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-15 02:54:14,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-15 02:54:14,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:14,329 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:14,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:54:14,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:54:14,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:54:14,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:14,330 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:14,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:54:14,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:54:14,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:14,331 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:14,331 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:54:14,331 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-15 02:54:14,331 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:54:14,331 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-15 02:54:14,331 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-15 02:54:14,331 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-15 02:54:14,331 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-15 02:54:14,332 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-15 02:54:14,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-15 02:54:14,331 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:14,332 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:14,332 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:14,332 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:14,332 DEBUG [Thread-864] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-15 02:54:14,332 DEBUG [Thread-864] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-15 02:54:14,332 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-15 02:54:14,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-15 02:54:14,411 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b7b06ed62cbe8eca9103c3d111564e6c, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:54:14,411 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34681] ipc.CallRunner(144): callId: 382 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:39132 deadline: 1681527314409, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b7b06ed62cbe8eca9103c3d111564e6c, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:14,412 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b7b06ed62cbe8eca9103c3d111564e6c, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:54:14,413 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34681] ipc.CallRunner(144): callId: 355 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:39106 deadline: 1681527314410, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b7b06ed62cbe8eca9103c3d111564e6c, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:14,432 DEBUG [Thread-864] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-15 02:54:14,433 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-15 02:54:14,434 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-15 02:54:14,435 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-15 02:54:14,435 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-15 02:54:14,435 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-15 02:54:14,435 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-15 02:54:14,435 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-15 02:54:14,435 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-15 02:54:14,436 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:14,436 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:54:14,436 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-15 02:54:14,436 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-15 02:54:14,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-15 02:54:14,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:54:14,437 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:14,437 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:14,437 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:14,437 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-15 02:54:14,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-15 02:54:14,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:54:14,437 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-15 02:54:14,438 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-15 02:54:14,438 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-15 02:54:14,438 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-15 02:54:14,438 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-15 02:54:14,438 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-15 02:54:14,442 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-15 02:54:14,442 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-15 02:54:14,442 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681527240182.b7b06ed62cbe8eca9103c3d111564e6c. 2023-04-15 02:54:14,442 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681527240182.b7b06ed62cbe8eca9103c3d111564e6c. started... 2023-04-15 02:54:14,442 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681527240182.b7b06ed62cbe8eca9103c3d111564e6c. as already flushing 2023-04-15 02:54:14,442 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681527240182.b7b06ed62cbe8eca9103c3d111564e6c. 2023-04-15 02:54:14,444 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-15 02:54:14,444 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-15 02:54:14,444 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:14,444 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-15 02:54:14,444 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34681,1681527209133' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-15 02:54:14,450 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:14,450 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:14,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:14,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:54:14,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:54:14,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:54:14,451 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:14,451 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-15 02:54:14,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:54:14,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:54:14,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:14,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:54:14,453 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34681,1681527209133' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-15 02:54:14,453 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@45e271bf[Count = 0] remaining members to acquire global barrier 2023-04-15 02:54:14,453 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-15 02:54:14,453 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:14,454 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:14,454 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:14,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:14,455 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:14,455 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-15 02:54:14,456 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-15 02:54:14,456 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-15 02:54:14,456 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34681,1681527209133' in zk 2023-04-15 02:54:14,457 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-15 02:54:14,457 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-15 02:54:14,457 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-15 02:54:14,457 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:14,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:14,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:54:14,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:54:14,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:54:14,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:54:14,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:54:14,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:14,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:54:14,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:54:14,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:14,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34681,1681527209133': 2023-04-15 02:54:14,461 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34681,1681527209133' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-15 02:54:14,461 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-15 02:54:14,461 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-15 02:54:14,461 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-15 02:54:14,461 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-15 02:54:14,461 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-15 02:54:14,463 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:14,464 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-15 02:54:14,464 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:14,464 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:14,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:14,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:54:14,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:54:14,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:14,464 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-15 02:54:14,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-15 02:54:14,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:14,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:54:14,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:54:14,465 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:14,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:54:14,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:54:14,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:14,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:54:14,466 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:14,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:54:14,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:14,468 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:14,468 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:54:14,468 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:54:14,468 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:14,468 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:14,468 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:14,468 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:14,468 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-15 02:54:14,469 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-15 02:54:14,469 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-15 02:54:14,469 DEBUG [Thread-864] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-15 02:54:14,469 DEBUG [Thread-864] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-15 02:54:14,469 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-15 02:54:14,469 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-15 02:54:14,469 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-15 02:54:14,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-15 02:54:14,469 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-15 02:54:14,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-15 02:54:14,504 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230415fb515c3897bc482a88c85a390a48b21e to hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230415fb515c3897bc482a88c85a390a48b21e 2023-04-15 02:54:14,569 DEBUG [Thread-864] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-15 02:54:14,570 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-15 02:54:14,571 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-15 02:54:14,573 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-15 02:54:14,573 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-15 02:54:14,573 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-15 02:54:14,573 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-15 02:54:14,573 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-15 02:54:14,574 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-15 02:54:14,574 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:14,574 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:54:14,575 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-15 02:54:14,576 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-15 02:54:14,575 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:14,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-15 02:54:14,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:54:14,576 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:14,576 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:14,576 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-15 02:54:14,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-15 02:54:14,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:54:14,577 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-15 02:54:14,577 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-15 02:54:14,577 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-15 02:54:14,577 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-15 02:54:14,577 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-15 02:54:14,577 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-15 02:54:14,578 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-15 02:54:14,578 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-15 02:54:14,578 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681527240182.b7b06ed62cbe8eca9103c3d111564e6c. 2023-04-15 02:54:14,578 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681527240182.b7b06ed62cbe8eca9103c3d111564e6c. started... 2023-04-15 02:54:14,578 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681527240182.b7b06ed62cbe8eca9103c3d111564e6c. as already flushing 2023-04-15 02:54:14,578 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681527240182.b7b06ed62cbe8eca9103c3d111564e6c. 2023-04-15 02:54:14,578 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-15 02:54:14,578 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-15 02:54:14,578 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:14,578 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-15 02:54:14,578 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34681,1681527209133' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-15 02:54:14,579 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:14,579 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:14,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:14,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:54:14,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:54:14,580 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:14,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:54:14,580 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-15 02:54:14,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:54:14,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:54:14,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:14,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:54:14,581 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34681,1681527209133' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-15 02:54:14,581 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3f029119[Count = 0] remaining members to acquire global barrier 2023-04-15 02:54:14,581 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-15 02:54:14,581 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:14,582 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:14,582 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:14,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:14,582 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-15 02:54:14,582 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:14,582 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-15 02:54:14,582 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-15 02:54:14,582 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34681,1681527209133' in zk 2023-04-15 02:54:14,583 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-15 02:54:14,583 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:14,583 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-15 02:54:14,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:14,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:54:14,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:54:14,583 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-15 02:54:14,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:54:14,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:54:14,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:54:14,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:14,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:54:14,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:54:14,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:14,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34681,1681527209133': 2023-04-15 02:54:14,586 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34681,1681527209133' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-15 02:54:14,586 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-15 02:54:14,586 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-15 02:54:14,586 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-15 02:54:14,586 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-15 02:54:14,586 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-15 02:54:14,587 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:14,588 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-15 02:54:14,588 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:14,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:14,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:54:14,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:54:14,588 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:14,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:14,588 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-15 02:54:14,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-15 02:54:14,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:54:14,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:14,588 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:14,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:54:14,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:54:14,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:54:14,590 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:14,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:14,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:54:14,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:54:14,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:14,592 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-15 02:54:14,592 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-15 02:54:14,592 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-15 02:54:14,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-15 02:54:14,592 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-15 02:54:14,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-15 02:54:14,592 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:14,593 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:54:14,593 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-15 02:54:14,593 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-15 02:54:14,593 DEBUG [Thread-864] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-15 02:54:14,593 DEBUG [Thread-864] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-15 02:54:14,593 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-15 02:54:14,593 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:54:14,593 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:14,593 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:14,593 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:14,593 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:14,593 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-15 02:54:14,693 DEBUG [Thread-864] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-15 02:54:14,693 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-15 02:54:14,695 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-15 02:54:14,695 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-15 02:54:14,695 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-15 02:54:14,695 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-15 02:54:14,696 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-15 02:54:14,696 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-15 02:54:14,696 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-15 02:54:14,696 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:14,696 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:54:14,697 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-15 02:54:14,697 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-15 02:54:14,697 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:14,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-15 02:54:14,697 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:14,697 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-15 02:54:14,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:54:14,697 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:14,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-15 02:54:14,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:54:14,697 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-15 02:54:14,698 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-15 02:54:14,698 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-15 02:54:14,698 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-15 02:54:14,698 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-15 02:54:14,698 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-15 02:54:14,698 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-15 02:54:14,698 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-15 02:54:14,698 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681527240182.b7b06ed62cbe8eca9103c3d111564e6c. 2023-04-15 02:54:14,698 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681527240182.b7b06ed62cbe8eca9103c3d111564e6c. started... 2023-04-15 02:54:14,698 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681527240182.b7b06ed62cbe8eca9103c3d111564e6c. as already flushing 2023-04-15 02:54:14,698 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681527240182.b7b06ed62cbe8eca9103c3d111564e6c. 2023-04-15 02:54:14,698 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-15 02:54:14,698 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-15 02:54:14,698 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:14,698 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-15 02:54:14,698 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34681,1681527209133' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-15 02:54:14,699 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:14,699 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:14,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:14,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:54:14,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:54:14,699 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:14,699 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-15 02:54:14,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:54:14,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:54:14,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:54:14,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:14,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:54:14,701 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34681,1681527209133' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-15 02:54:14,701 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-15 02:54:14,701 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@386635af[Count = 0] remaining members to acquire global barrier 2023-04-15 02:54:14,701 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:14,702 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:14,702 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:14,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:14,702 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-15 02:54:14,702 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-15 02:54:14,702 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34681,1681527209133' in zk 2023-04-15 02:54:14,702 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:14,702 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-15 02:54:14,703 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-15 02:54:14,703 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-15 02:54:14,703 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-15 02:54:14,707 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:14,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:14,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:54:14,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:54:14,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:54:14,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:54:14,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:54:14,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:14,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:54:14,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:54:14,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:14,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34681,1681527209133': 2023-04-15 02:54:14,712 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34681,1681527209133' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-15 02:54:14,712 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-15 02:54:14,712 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-15 02:54:14,712 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-15 02:54:14,712 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-15 02:54:14,712 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-15 02:54:14,715 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b7b06ed62cbe8eca9103c3d111564e6c, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:54:14,715 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34681] ipc.CallRunner(144): callId: 357 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:39106 deadline: 1681527314715, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b7b06ed62cbe8eca9103c3d111564e6c, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:14,716 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b7b06ed62cbe8eca9103c3d111564e6c, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:54:14,716 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34681] ipc.CallRunner(144): callId: 384 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:39132 deadline: 1681527314715, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b7b06ed62cbe8eca9103c3d111564e6c, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:14,717 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:14,717 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-15 02:54:14,717 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:14,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:14,717 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:14,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:14,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:54:14,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:54:14,718 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-15 02:54:14,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-15 02:54:14,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:54:14,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:14,718 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:14,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:54:14,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:54:14,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:54:14,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:14,719 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:14,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:54:14,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:54:14,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:14,721 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-15 02:54:14,721 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-15 02:54:14,721 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-15 02:54:14,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-15 02:54:14,721 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:14,721 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:54:14,721 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:54:14,721 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:14,721 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:14,721 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:14,721 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-15 02:54:14,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-15 02:54:14,721 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-15 02:54:14,722 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-15 02:54:14,721 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-15 02:54:14,721 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:14,723 DEBUG [Thread-864] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-15 02:54:14,723 DEBUG [Thread-864] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-15 02:54:14,824 DEBUG [Thread-864] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-15 02:54:14,824 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-15 02:54:14,825 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-15 02:54:14,826 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-15 02:54:14,826 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-15 02:54:14,827 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-15 02:54:14,827 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-15 02:54:14,827 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-15 02:54:14,827 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-15 02:54:14,827 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:14,827 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:54:14,828 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-15 02:54:14,828 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:14,828 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-15 02:54:14,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-15 02:54:14,828 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:14,828 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-15 02:54:14,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:54:14,828 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:14,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-15 02:54:14,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:54:14,828 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-15 02:54:14,829 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-15 02:54:14,829 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-15 02:54:14,829 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-15 02:54:14,829 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-15 02:54:14,829 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-15 02:54:14,829 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-15 02:54:14,829 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-15 02:54:14,829 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681527240182.b7b06ed62cbe8eca9103c3d111564e6c. 2023-04-15 02:54:14,829 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681527240182.b7b06ed62cbe8eca9103c3d111564e6c. started... 2023-04-15 02:54:14,829 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681527240182.b7b06ed62cbe8eca9103c3d111564e6c. as already flushing 2023-04-15 02:54:14,829 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681527240182.b7b06ed62cbe8eca9103c3d111564e6c. 2023-04-15 02:54:14,829 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-15 02:54:14,830 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-15 02:54:14,830 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:14,830 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-15 02:54:14,830 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34681,1681527209133' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-15 02:54:14,830 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:14,830 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:14,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:14,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:54:14,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:54:14,831 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:14,831 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-15 02:54:14,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:54:14,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:54:14,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:54:14,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:14,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:54:14,832 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34681,1681527209133' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-15 02:54:14,832 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-15 02:54:14,832 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1af544b5[Count = 0] remaining members to acquire global barrier 2023-04-15 02:54:14,832 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:14,833 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:14,833 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:14,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:14,833 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-15 02:54:14,833 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:14,833 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-15 02:54:14,833 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-15 02:54:14,833 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34681,1681527209133' in zk 2023-04-15 02:54:14,834 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:14,834 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-15 02:54:14,834 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-15 02:54:14,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:14,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:54:14,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:54:14,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:54:14,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:54:14,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:54:14,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:14,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:54:14,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:54:14,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:14,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34681,1681527209133': 2023-04-15 02:54:14,836 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34681,1681527209133' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-15 02:54:14,834 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-15 02:54:14,836 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-15 02:54:14,838 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-15 02:54:14,838 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-15 02:54:14,838 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-15 02:54:14,838 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-15 02:54:14,839 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:14,839 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:14,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:14,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:54:14,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:54:14,839 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-15 02:54:14,839 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:14,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:14,840 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-15 02:54:14,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-15 02:54:14,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:54:14,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:14,840 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:14,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:54:14,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:54:14,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:54:14,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:14,841 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:14,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:54:14,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:54:14,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:14,842 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:14,842 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-15 02:54:14,842 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-15 02:54:14,842 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:54:14,843 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:54:14,843 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:14,843 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:14,843 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:14,843 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:14,843 DEBUG [Thread-864] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-15 02:54:14,843 DEBUG [Thread-864] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-15 02:54:14,843 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-15 02:54:14,842 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-15 02:54:14,843 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-15 02:54:14,843 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-15 02:54:14,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-15 02:54:14,843 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-15 02:54:14,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-15 02:54:14,920 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=669, memsize=42.5 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/b7b06ed62cbe8eca9103c3d111564e6c/.tmp/A/b745ac63855c4da3bdb4f690d537cf6a 2023-04-15 02:54:14,943 DEBUG [Thread-864] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-15 02:54:14,944 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-15 02:54:14,945 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=669 (bloomFilter=true), to=hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/b7b06ed62cbe8eca9103c3d111564e6c/.tmp/B/4813d52cc442437d876f752f5e126c2a 2023-04-15 02:54:14,946 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-15 02:54:14,946 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-15 02:54:14,946 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-15 02:54:14,947 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-15 02:54:14,947 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-15 02:54:14,947 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-15 02:54:14,947 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-15 02:54:14,947 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:14,947 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:54:14,948 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:14,948 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-15 02:54:14,948 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-15 02:54:14,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-15 02:54:14,948 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:14,948 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-15 02:54:14,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:54:14,949 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:14,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-15 02:54:14,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:54:14,949 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-15 02:54:14,949 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-15 02:54:14,950 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-15 02:54:14,950 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-15 02:54:14,950 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-15 02:54:14,950 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-15 02:54:14,950 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-15 02:54:14,950 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-15 02:54:14,950 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681527240182.b7b06ed62cbe8eca9103c3d111564e6c. 2023-04-15 02:54:14,950 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681527240182.b7b06ed62cbe8eca9103c3d111564e6c. started... 2023-04-15 02:54:14,950 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681527240182.b7b06ed62cbe8eca9103c3d111564e6c. as already flushing 2023-04-15 02:54:14,950 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681527240182.b7b06ed62cbe8eca9103c3d111564e6c. 2023-04-15 02:54:14,950 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-15 02:54:14,950 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-15 02:54:14,950 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:14,950 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-15 02:54:14,950 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34681,1681527209133' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-15 02:54:14,951 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:14,951 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:14,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:14,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:54:14,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:54:14,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:54:14,952 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:14,952 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-15 02:54:14,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:54:14,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:54:14,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:14,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:54:14,953 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34681,1681527209133' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-15 02:54:14,953 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4c4ca1dd[Count = 0] remaining members to acquire global barrier 2023-04-15 02:54:14,953 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-15 02:54:14,953 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:14,954 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:14,954 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-15 02:54:14,954 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:14,954 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:14,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:14,955 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-15 02:54:14,955 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-15 02:54:14,955 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34681,1681527209133' in zk 2023-04-15 02:54:14,955 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:14,955 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-15 02:54:14,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:14,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:54:14,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:54:14,955 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-15 02:54:14,956 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-15 02:54:14,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:54:14,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:54:14,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:54:14,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:14,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:54:14,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:54:14,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:14,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34681,1681527209133': 2023-04-15 02:54:14,958 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34681,1681527209133' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-15 02:54:14,958 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-15 02:54:14,958 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-15 02:54:14,958 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-15 02:54:14,958 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-15 02:54:14,958 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-15 02:54:14,959 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:14,959 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-15 02:54:14,959 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:14,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:14,959 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-15 02:54:14,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-15 02:54:14,959 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:14,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:14,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:54:14,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:54:14,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:14,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:54:14,960 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=669 (bloomFilter=true), to=hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/b7b06ed62cbe8eca9103c3d111564e6c/.tmp/C/1f3a607057f74152bacf31d561dd5835 2023-04-15 02:54:14,960 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:14,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:54:14,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:54:14,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:54:14,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:14,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:54:14,963 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:14,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:54:14,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:14,971 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:14,971 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:54:14,971 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:54:14,971 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:14,971 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:14,971 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:14,971 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:14,972 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-15 02:54:14,972 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-15 02:54:14,972 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-15 02:54:14,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-15 02:54:14,972 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-15 02:54:14,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-15 02:54:14,974 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-15 02:54:14,974 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-15 02:54:14,974 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-15 02:54:14,974 DEBUG [Thread-864] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-15 02:54:14,974 DEBUG [Thread-864] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-15 02:54:14,975 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/b7b06ed62cbe8eca9103c3d111564e6c/.tmp/A/b745ac63855c4da3bdb4f690d537cf6a as hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/b7b06ed62cbe8eca9103c3d111564e6c/A/b745ac63855c4da3bdb4f690d537cf6a 2023-04-15 02:54:14,980 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/b7b06ed62cbe8eca9103c3d111564e6c/A/b745ac63855c4da3bdb4f690d537cf6a, entries=250, sequenceid=669, filesize=39.2 K 2023-04-15 02:54:14,981 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/b7b06ed62cbe8eca9103c3d111564e6c/.tmp/B/4813d52cc442437d876f752f5e126c2a as hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/b7b06ed62cbe8eca9103c3d111564e6c/B/4813d52cc442437d876f752f5e126c2a 2023-04-15 02:54:15,012 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/b7b06ed62cbe8eca9103c3d111564e6c/B/4813d52cc442437d876f752f5e126c2a, entries=150, sequenceid=669, filesize=11.9 K 2023-04-15 02:54:15,014 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/b7b06ed62cbe8eca9103c3d111564e6c/.tmp/C/1f3a607057f74152bacf31d561dd5835 as hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/b7b06ed62cbe8eca9103c3d111564e6c/C/1f3a607057f74152bacf31d561dd5835 2023-04-15 02:54:15,018 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/b7b06ed62cbe8eca9103c3d111564e6c/C/1f3a607057f74152bacf31d561dd5835, entries=150, sequenceid=669, filesize=11.9 K 2023-04-15 02:54:15,019 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~127.47 KB/130530, heapSize ~348.61 KB/356976, currentSize=60.38 KB/61830 for b7b06ed62cbe8eca9103c3d111564e6c in 939ms, sequenceid=669, compaction requested=true 2023-04-15 02:54:15,019 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for b7b06ed62cbe8eca9103c3d111564e6c: 2023-04-15 02:54:15,019 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-15 02:54:15,019 DEBUG [RS:0;jenkins-hbase20:34681-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-15 02:54:15,019 DEBUG [RS:0;jenkins-hbase20:34681-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-15 02:54:15,019 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-15 02:54:15,019 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-15 02:54:15,021 DEBUG [RS:0;jenkins-hbase20:34681-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 118190 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-15 02:54:15,021 DEBUG [RS:0;jenkins-hbase20:34681-shortCompactions-0] regionserver.HStore(1912): b7b06ed62cbe8eca9103c3d111564e6c/A is initiating minor compaction (all files) 2023-04-15 02:54:15,021 INFO [RS:0;jenkins-hbase20:34681-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of b7b06ed62cbe8eca9103c3d111564e6c/A in TestAcidGuarantees,,1681527240182.b7b06ed62cbe8eca9103c3d111564e6c. 2023-04-15 02:54:15,021 INFO [RS:0;jenkins-hbase20:34681-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/b7b06ed62cbe8eca9103c3d111564e6c/A/f789c3b8ecdb49d6ac6ba9ca495a5b11, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/b7b06ed62cbe8eca9103c3d111564e6c/A/6bd682223d1542898d764194cf4d3380, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/b7b06ed62cbe8eca9103c3d111564e6c/A/acdbc16f0bd4470e853d39d9237b4be9, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/b7b06ed62cbe8eca9103c3d111564e6c/A/b745ac63855c4da3bdb4f690d537cf6a] into tmpdir=hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/b7b06ed62cbe8eca9103c3d111564e6c/.tmp, totalSize=115.4 K 2023-04-15 02:54:15,021 DEBUG [RS:0;jenkins-hbase20:34681-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 46276 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-15 02:54:15,021 DEBUG [RS:0;jenkins-hbase20:34681-longCompactions-0] regionserver.HStore(1912): b7b06ed62cbe8eca9103c3d111564e6c/B is initiating minor compaction (all files) 2023-04-15 02:54:15,021 INFO [RS:0;jenkins-hbase20:34681-longCompactions-0] regionserver.HRegion(2259): Starting compaction of b7b06ed62cbe8eca9103c3d111564e6c/B in TestAcidGuarantees,,1681527240182.b7b06ed62cbe8eca9103c3d111564e6c. 2023-04-15 02:54:15,022 INFO [RS:0;jenkins-hbase20:34681-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/b7b06ed62cbe8eca9103c3d111564e6c/B/e63856d5116b41328e79561838e6068c, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/b7b06ed62cbe8eca9103c3d111564e6c/B/7e797ffd65c5478e80443ace7c715fb7, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/b7b06ed62cbe8eca9103c3d111564e6c/B/860fa6f1f1514f058db8f8453e5c1fc7, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/b7b06ed62cbe8eca9103c3d111564e6c/B/4813d52cc442437d876f752f5e126c2a] into tmpdir=hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/b7b06ed62cbe8eca9103c3d111564e6c/.tmp, totalSize=45.2 K 2023-04-15 02:54:15,022 DEBUG [RS:0;jenkins-hbase20:34681-shortCompactions-0] compactions.Compactor(207): Compacting f789c3b8ecdb49d6ac6ba9ca495a5b11, keycount=150, bloomtype=ROW, size=25.7 K, encoding=NONE, compression=NONE, seqNum=177, earliestPutTs=1681527250839 2023-04-15 02:54:15,022 DEBUG [RS:0;jenkins-hbase20:34681-shortCompactions-0] compactions.Compactor(207): Compacting 6bd682223d1542898d764194cf4d3380, keycount=150, bloomtype=ROW, size=25.3 K, encoding=NONE, compression=NONE, seqNum=199, earliestPutTs=1681527251008 2023-04-15 02:54:15,023 DEBUG [RS:0;jenkins-hbase20:34681-longCompactions-0] compactions.Compactor(207): Compacting e63856d5116b41328e79561838e6068c, keycount=150, bloomtype=ROW, size=12.1 K, encoding=NONE, compression=NONE, seqNum=177, earliestPutTs=1681527250839 2023-04-15 02:54:15,024 DEBUG [RS:0;jenkins-hbase20:34681-shortCompactions-0] compactions.Compactor(207): Compacting acdbc16f0bd4470e853d39d9237b4be9, keycount=150, bloomtype=ROW, size=25.3 K, encoding=NONE, compression=NONE, seqNum=207, earliestPutTs=1681527253145 2023-04-15 02:54:15,024 DEBUG [RS:0;jenkins-hbase20:34681-shortCompactions-0] compactions.Compactor(207): Compacting b745ac63855c4da3bdb4f690d537cf6a, keycount=250, bloomtype=ROW, size=39.2 K, encoding=NONE, compression=NONE, seqNum=669, earliestPutTs=1681527254065 2023-04-15 02:54:15,025 DEBUG [RS:0;jenkins-hbase20:34681-longCompactions-0] compactions.Compactor(207): Compacting 7e797ffd65c5478e80443ace7c715fb7, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=199, earliestPutTs=1681527251008 2023-04-15 02:54:15,025 DEBUG [RS:0;jenkins-hbase20:34681-longCompactions-0] compactions.Compactor(207): Compacting 860fa6f1f1514f058db8f8453e5c1fc7, keycount=100, bloomtype=ROW, size=9.4 K, encoding=NONE, compression=NONE, seqNum=207, earliestPutTs=1681527253145 2023-04-15 02:54:15,027 DEBUG [RS:0;jenkins-hbase20:34681-longCompactions-0] compactions.Compactor(207): Compacting 4813d52cc442437d876f752f5e126c2a, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=669, earliestPutTs=1681527254073 2023-04-15 02:54:15,063 INFO [RS:0;jenkins-hbase20:34681-longCompactions-0] throttle.PressureAwareThroughputController(145): b7b06ed62cbe8eca9103c3d111564e6c#B#compaction#147 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-15 02:54:15,074 DEBUG [Thread-864] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-15 02:54:15,075 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-15 02:54:15,077 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-15 02:54:15,090 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-15 02:54:15,090 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-15 02:54:15,090 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-15 02:54:15,090 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-15 02:54:15,090 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-15 02:54:15,090 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-15 02:54:15,091 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:15,091 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:54:15,091 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:15,091 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-15 02:54:15,091 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-15 02:54:15,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-15 02:54:15,092 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:15,092 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-15 02:54:15,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:54:15,092 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:15,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-15 02:54:15,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:54:15,093 INFO [RS:0;jenkins-hbase20:34681-shortCompactions-0] throttle.PressureAwareThroughputController(145): b7b06ed62cbe8eca9103c3d111564e6c#A#compaction#146 average throughput is 0.51 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-15 02:54:15,093 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-15 02:54:15,106 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-15 02:54:15,106 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-15 02:54:15,106 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-15 02:54:15,106 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-15 02:54:15,106 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-15 02:54:15,108 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-15 02:54:15,108 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-15 02:54:15,108 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681527240182.b7b06ed62cbe8eca9103c3d111564e6c. 2023-04-15 02:54:15,108 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681527240182.b7b06ed62cbe8eca9103c3d111564e6c. started... 2023-04-15 02:54:15,108 INFO [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing b7b06ed62cbe8eca9103c3d111564e6c 3/3 column families, dataSize=60.38 KB heapSize=180.05 KB 2023-04-15 02:54:15,108 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK b7b06ed62cbe8eca9103c3d111564e6c, store=A 2023-04-15 02:54:15,108 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-15 02:54:15,109 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK b7b06ed62cbe8eca9103c3d111564e6c, store=B 2023-04-15 02:54:15,109 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-15 02:54:15,109 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK b7b06ed62cbe8eca9103c3d111564e6c, store=C 2023-04-15 02:54:15,109 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-15 02:54:15,220 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681527240182.b7b06ed62cbe8eca9103c3d111564e6c. as already flushing 2023-04-15 02:54:15,220 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34681] regionserver.HRegion(9158): Flush requested on b7b06ed62cbe8eca9103c3d111564e6c 2023-04-15 02:54:15,270 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b7b06ed62cbe8eca9103c3d111564e6c, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:54:15,270 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34681] ipc.CallRunner(144): callId: 368 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:39106 deadline: 1681527315266, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b7b06ed62cbe8eca9103c3d111564e6c, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:15,271 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b7b06ed62cbe8eca9103c3d111564e6c, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:54:15,271 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34681] ipc.CallRunner(144): callId: 394 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:39132 deadline: 1681527315266, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b7b06ed62cbe8eca9103c3d111564e6c, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:15,374 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b7b06ed62cbe8eca9103c3d111564e6c, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:54:15,375 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34681] ipc.CallRunner(144): callId: 370 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:39106 deadline: 1681527315371, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b7b06ed62cbe8eca9103c3d111564e6c, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:15,375 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b7b06ed62cbe8eca9103c3d111564e6c, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:54:15,375 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34681] ipc.CallRunner(144): callId: 396 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:39132 deadline: 1681527315371, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b7b06ed62cbe8eca9103c3d111564e6c, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:15,504 DEBUG [RS:0;jenkins-hbase20:34681-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/b7b06ed62cbe8eca9103c3d111564e6c/.tmp/B/9168c283c4e345d9b692f4ad877325c0 as hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/b7b06ed62cbe8eca9103c3d111564e6c/B/9168c283c4e345d9b692f4ad877325c0 2023-04-15 02:54:15,511 INFO [RS:0;jenkins-hbase20:34681-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in b7b06ed62cbe8eca9103c3d111564e6c/B of b7b06ed62cbe8eca9103c3d111564e6c into 9168c283c4e345d9b692f4ad877325c0(size=12.4 K), total size for store is 12.4 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-15 02:54:15,511 DEBUG [RS:0;jenkins-hbase20:34681-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for b7b06ed62cbe8eca9103c3d111564e6c: 2023-04-15 02:54:15,511 INFO [RS:0;jenkins-hbase20:34681-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681527240182.b7b06ed62cbe8eca9103c3d111564e6c., storeName=b7b06ed62cbe8eca9103c3d111564e6c/B, priority=12, startTime=1681527255019; duration=0sec 2023-04-15 02:54:15,511 DEBUG [RS:0;jenkins-hbase20:34681-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-15 02:54:15,511 DEBUG [RS:0;jenkins-hbase20:34681-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-04-15 02:54:15,512 DEBUG [RS:0;jenkins-hbase20:34681-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 58244 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-04-15 02:54:15,513 DEBUG [RS:0;jenkins-hbase20:34681-longCompactions-0] regionserver.HStore(1912): b7b06ed62cbe8eca9103c3d111564e6c/C is initiating minor compaction (all files) 2023-04-15 02:54:15,513 INFO [RS:0;jenkins-hbase20:34681-longCompactions-0] regionserver.HRegion(2259): Starting compaction of b7b06ed62cbe8eca9103c3d111564e6c/C in TestAcidGuarantees,,1681527240182.b7b06ed62cbe8eca9103c3d111564e6c. 2023-04-15 02:54:15,513 INFO [RS:0;jenkins-hbase20:34681-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/b7b06ed62cbe8eca9103c3d111564e6c/C/a1fdb8330b4f401484a795f4913c211b, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/b7b06ed62cbe8eca9103c3d111564e6c/C/721c2abb2f524608bf5f421c3d3f9656, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/b7b06ed62cbe8eca9103c3d111564e6c/C/cd3f2f0d61b740ebbce4c8787269d3b1, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/b7b06ed62cbe8eca9103c3d111564e6c/C/6630954ef78e4fa2a6911d94d6b1daba, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/b7b06ed62cbe8eca9103c3d111564e6c/C/1f3a607057f74152bacf31d561dd5835] into tmpdir=hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/b7b06ed62cbe8eca9103c3d111564e6c/.tmp, totalSize=56.9 K 2023-04-15 02:54:15,514 DEBUG [RS:0;jenkins-hbase20:34681-longCompactions-0] compactions.Compactor(207): Compacting a1fdb8330b4f401484a795f4913c211b, keycount=150, bloomtype=ROW, size=12.1 K, encoding=NONE, compression=NONE, seqNum=163, earliestPutTs=1681527249855 2023-04-15 02:54:15,515 DEBUG [RS:0;jenkins-hbase20:34681-longCompactions-0] compactions.Compactor(207): Compacting 721c2abb2f524608bf5f421c3d3f9656, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=177, earliestPutTs=1681527250839 2023-04-15 02:54:15,515 DEBUG [RS:0;jenkins-hbase20:34681-longCompactions-0] compactions.Compactor(207): Compacting cd3f2f0d61b740ebbce4c8787269d3b1, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=199, earliestPutTs=1681527251008 2023-04-15 02:54:15,515 DEBUG [RS:0;jenkins-hbase20:34681-longCompactions-0] compactions.Compactor(207): Compacting 6630954ef78e4fa2a6911d94d6b1daba, keycount=100, bloomtype=ROW, size=9.4 K, encoding=NONE, compression=NONE, seqNum=207, earliestPutTs=1681527253145 2023-04-15 02:54:15,515 DEBUG [RS:0;jenkins-hbase20:34681-longCompactions-0] compactions.Compactor(207): Compacting 1f3a607057f74152bacf31d561dd5835, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=669, earliestPutTs=1681527254073 2023-04-15 02:54:15,527 INFO [RS:0;jenkins-hbase20:34681-longCompactions-0] throttle.PressureAwareThroughputController(145): b7b06ed62cbe8eca9103c3d111564e6c#C#compaction#149 average throughput is 2.18 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-15 02:54:15,539 DEBUG [RS:0;jenkins-hbase20:34681-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/b7b06ed62cbe8eca9103c3d111564e6c/.tmp/C/24768a1fbb7943f48b50e45cb17b549e as hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/b7b06ed62cbe8eca9103c3d111564e6c/C/24768a1fbb7943f48b50e45cb17b549e 2023-04-15 02:54:15,544 INFO [RS:0;jenkins-hbase20:34681-longCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in b7b06ed62cbe8eca9103c3d111564e6c/C of b7b06ed62cbe8eca9103c3d111564e6c into 24768a1fbb7943f48b50e45cb17b549e(size=12.4 K), total size for store is 12.4 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-15 02:54:15,544 DEBUG [RS:0;jenkins-hbase20:34681-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for b7b06ed62cbe8eca9103c3d111564e6c: 2023-04-15 02:54:15,544 INFO [RS:0;jenkins-hbase20:34681-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681527240182.b7b06ed62cbe8eca9103c3d111564e6c., storeName=b7b06ed62cbe8eca9103c3d111564e6c/C, priority=11, startTime=1681527255019; duration=0sec 2023-04-15 02:54:15,544 DEBUG [RS:0;jenkins-hbase20:34681-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-15 02:54:15,574 DEBUG [RS:0;jenkins-hbase20:34681-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/b7b06ed62cbe8eca9103c3d111564e6c/.tmp/A/e8204d9538ec45a184fb961fe5527933 as hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/b7b06ed62cbe8eca9103c3d111564e6c/A/e8204d9538ec45a184fb961fe5527933 2023-04-15 02:54:15,576 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b7b06ed62cbe8eca9103c3d111564e6c, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:54:15,576 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34681] ipc.CallRunner(144): callId: 398 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:39132 deadline: 1681527315576, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b7b06ed62cbe8eca9103c3d111564e6c, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:15,577 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b7b06ed62cbe8eca9103c3d111564e6c, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:54:15,578 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34681] ipc.CallRunner(144): callId: 372 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:39106 deadline: 1681527315577, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b7b06ed62cbe8eca9103c3d111564e6c, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:15,582 INFO [RS:0;jenkins-hbase20:34681-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in b7b06ed62cbe8eca9103c3d111564e6c/A of b7b06ed62cbe8eca9103c3d111564e6c into e8204d9538ec45a184fb961fe5527933(size=25.9 K), total size for store is 25.9 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-15 02:54:15,582 DEBUG [RS:0;jenkins-hbase20:34681-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for b7b06ed62cbe8eca9103c3d111564e6c: 2023-04-15 02:54:15,582 INFO [RS:0;jenkins-hbase20:34681-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681527240182.b7b06ed62cbe8eca9103c3d111564e6c., storeName=b7b06ed62cbe8eca9103c3d111564e6c/A, priority=12, startTime=1681527255019; duration=0sec 2023-04-15 02:54:15,583 DEBUG [RS:0;jenkins-hbase20:34681-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-15 02:54:15,584 INFO [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230415fe29b74e7a024e8198d854a3dca04cd2 to hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230415fe29b74e7a024e8198d854a3dca04cd2 2023-04-15 02:54:15,879 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b7b06ed62cbe8eca9103c3d111564e6c, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:54:15,879 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34681] ipc.CallRunner(144): callId: 374 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:39106 deadline: 1681527315879, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b7b06ed62cbe8eca9103c3d111564e6c, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:15,880 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b7b06ed62cbe8eca9103c3d111564e6c, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:54:15,880 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34681] ipc.CallRunner(144): callId: 400 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:39132 deadline: 1681527315879, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b7b06ed62cbe8eca9103c3d111564e6c, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:15,990 INFO [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=681, memsize=20.1 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/b7b06ed62cbe8eca9103c3d111564e6c/.tmp/A/dcda3bbd8a00469a96653f91fa83ddad 2023-04-15 02:54:16,385 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b7b06ed62cbe8eca9103c3d111564e6c, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:54:16,385 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34681] ipc.CallRunner(144): callId: 402 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:39132 deadline: 1681527316383, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b7b06ed62cbe8eca9103c3d111564e6c, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:16,385 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b7b06ed62cbe8eca9103c3d111564e6c, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:54:16,385 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34681] ipc.CallRunner(144): callId: 376 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:39106 deadline: 1681527316384, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b7b06ed62cbe8eca9103c3d111564e6c, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:16,412 INFO [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=681 (bloomFilter=true), to=hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/b7b06ed62cbe8eca9103c3d111564e6c/.tmp/B/51687ca44ed04e20b7384f9f2e53d0da 2023-04-15 02:54:16,823 INFO [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=681 (bloomFilter=true), to=hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/b7b06ed62cbe8eca9103c3d111564e6c/.tmp/C/4b3fbff5251742bbb1852d90743ca06f 2023-04-15 02:54:16,827 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/b7b06ed62cbe8eca9103c3d111564e6c/.tmp/A/dcda3bbd8a00469a96653f91fa83ddad as hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/b7b06ed62cbe8eca9103c3d111564e6c/A/dcda3bbd8a00469a96653f91fa83ddad 2023-04-15 02:54:16,831 INFO [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/b7b06ed62cbe8eca9103c3d111564e6c/A/dcda3bbd8a00469a96653f91fa83ddad, entries=150, sequenceid=681, filesize=25.4 K 2023-04-15 02:54:16,832 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/b7b06ed62cbe8eca9103c3d111564e6c/.tmp/B/51687ca44ed04e20b7384f9f2e53d0da as hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/b7b06ed62cbe8eca9103c3d111564e6c/B/51687ca44ed04e20b7384f9f2e53d0da 2023-04-15 02:54:16,835 INFO [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/b7b06ed62cbe8eca9103c3d111564e6c/B/51687ca44ed04e20b7384f9f2e53d0da, entries=150, sequenceid=681, filesize=11.9 K 2023-04-15 02:54:16,836 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/b7b06ed62cbe8eca9103c3d111564e6c/.tmp/C/4b3fbff5251742bbb1852d90743ca06f as hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/b7b06ed62cbe8eca9103c3d111564e6c/C/4b3fbff5251742bbb1852d90743ca06f 2023-04-15 02:54:16,840 INFO [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/b7b06ed62cbe8eca9103c3d111564e6c/C/4b3fbff5251742bbb1852d90743ca06f, entries=150, sequenceid=681, filesize=11.9 K 2023-04-15 02:54:16,840 INFO [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~60.38 KB/61830, heapSize ~180 KB/184320, currentSize=114.05 KB/116790 for b7b06ed62cbe8eca9103c3d111564e6c in 1732ms, sequenceid=681, compaction requested=false 2023-04-15 02:54:16,841 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for b7b06ed62cbe8eca9103c3d111564e6c: 2023-04-15 02:54:16,841 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681527240182.b7b06ed62cbe8eca9103c3d111564e6c. 2023-04-15 02:54:16,841 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-15 02:54:16,841 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-15 02:54:16,841 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:16,841 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-15 02:54:16,841 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34681,1681527209133' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-15 02:54:16,842 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:16,842 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:16,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:16,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:54:16,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:54:16,842 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:16,842 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-15 02:54:16,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:54:16,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:54:16,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:54:16,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:16,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:54:16,844 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34681,1681527209133' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-15 02:54:16,844 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@67ed6b14[Count = 0] remaining members to acquire global barrier 2023-04-15 02:54:16,844 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-15 02:54:16,844 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:16,844 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:16,844 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:16,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:16,844 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-15 02:54:16,844 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-15 02:54:16,844 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34681,1681527209133' in zk 2023-04-15 02:54:16,844 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:16,845 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-15 02:54:16,846 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:16,846 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-15 02:54:16,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:16,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:54:16,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:54:16,846 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-15 02:54:16,846 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-15 02:54:16,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:54:16,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:54:16,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:54:16,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:16,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:54:16,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:54:16,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:16,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34681,1681527209133': 2023-04-15 02:54:16,848 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34681,1681527209133' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-15 02:54:16,848 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-15 02:54:16,848 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-15 02:54:16,849 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-15 02:54:16,849 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-15 02:54:16,849 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-15 02:54:16,849 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:16,849 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:16,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:16,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:54:16,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:54:16,850 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:16,850 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-15 02:54:16,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:16,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:54:16,850 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-15 02:54:16,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-15 02:54:16,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:54:16,850 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:16,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:16,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:54:16,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:54:16,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:16,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:54:16,851 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:16,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:54:16,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:16,852 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:16,852 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-15 02:54:16,852 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:54:16,852 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-15 02:54:16,852 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-15 02:54:16,852 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:54:16,853 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-15 02:54:16,853 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-15 02:54:16,853 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-15 02:54:16,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-15 02:54:16,853 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:16,853 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:16,853 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:16,853 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:16,853 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-15 02:54:16,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-15 02:54:16,853 DEBUG [Thread-864] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-15 02:54:16,853 DEBUG [Thread-864] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-15 02:54:16,953 DEBUG [Thread-864] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-15 02:54:16,954 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-15 02:54:16,955 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-15 02:54:16,956 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-15 02:54:16,956 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-15 02:54:16,956 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-15 02:54:16,956 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-15 02:54:16,956 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-15 02:54:16,956 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-15 02:54:16,956 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:16,956 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:54:16,957 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-15 02:54:16,957 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:16,957 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-15 02:54:16,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-15 02:54:16,957 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:16,957 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-15 02:54:16,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:54:16,958 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:16,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-15 02:54:16,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:54:16,958 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-15 02:54:16,958 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-15 02:54:16,958 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-15 02:54:16,958 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-15 02:54:16,958 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-15 02:54:16,958 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-15 02:54:16,958 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-15 02:54:16,958 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-15 02:54:16,958 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681527240182.b7b06ed62cbe8eca9103c3d111564e6c. 2023-04-15 02:54:16,959 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681527240182.b7b06ed62cbe8eca9103c3d111564e6c. started... 2023-04-15 02:54:16,959 INFO [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing b7b06ed62cbe8eca9103c3d111564e6c 3/3 column families, dataSize=114.05 KB heapSize=339.42 KB 2023-04-15 02:54:16,959 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK b7b06ed62cbe8eca9103c3d111564e6c, store=A 2023-04-15 02:54:16,959 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-15 02:54:16,959 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK b7b06ed62cbe8eca9103c3d111564e6c, store=B 2023-04-15 02:54:16,959 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-15 02:54:16,959 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK b7b06ed62cbe8eca9103c3d111564e6c, store=C 2023-04-15 02:54:16,959 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-15 02:54:17,370 INFO [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230415c1c28ef1b35541eba6007526680031cc to hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230415c1c28ef1b35541eba6007526680031cc 2023-04-15 02:54:17,376 INFO [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=704, memsize=38.0 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/b7b06ed62cbe8eca9103c3d111564e6c/.tmp/A/2b87e81781d04ac4aebf8b72abb33b61 2023-04-15 02:54:17,388 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34681] regionserver.HRegion(9158): Flush requested on b7b06ed62cbe8eca9103c3d111564e6c 2023-04-15 02:54:17,388 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681527240182.b7b06ed62cbe8eca9103c3d111564e6c. as already flushing 2023-04-15 02:54:17,411 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b7b06ed62cbe8eca9103c3d111564e6c, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:54:17,411 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34681] ipc.CallRunner(144): callId: 383 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:39106 deadline: 1681527317410, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b7b06ed62cbe8eca9103c3d111564e6c, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:17,411 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b7b06ed62cbe8eca9103c3d111564e6c, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:54:17,411 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34681] ipc.CallRunner(144): callId: 408 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:39132 deadline: 1681527317411, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b7b06ed62cbe8eca9103c3d111564e6c, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:17,512 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b7b06ed62cbe8eca9103c3d111564e6c, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:54:17,512 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34681] ipc.CallRunner(144): callId: 385 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:39106 deadline: 1681527317512, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b7b06ed62cbe8eca9103c3d111564e6c, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:17,513 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b7b06ed62cbe8eca9103c3d111564e6c, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:54:17,513 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34681] ipc.CallRunner(144): callId: 410 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:39132 deadline: 1681527317512, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b7b06ed62cbe8eca9103c3d111564e6c, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:17,716 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b7b06ed62cbe8eca9103c3d111564e6c, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:54:17,716 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34681] ipc.CallRunner(144): callId: 387 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:39106 deadline: 1681527317713, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b7b06ed62cbe8eca9103c3d111564e6c, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:17,716 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b7b06ed62cbe8eca9103c3d111564e6c, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:54:17,717 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34681] ipc.CallRunner(144): callId: 412 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:39132 deadline: 1681527317714, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b7b06ed62cbe8eca9103c3d111564e6c, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:17,790 INFO [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=704 (bloomFilter=true), to=hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/b7b06ed62cbe8eca9103c3d111564e6c/.tmp/B/cd61e6be1fc947dcaee243656edb94cc 2023-04-15 02:54:18,018 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b7b06ed62cbe8eca9103c3d111564e6c, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:54:18,018 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34681] ipc.CallRunner(144): callId: 414 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:39132 deadline: 1681527318017, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b7b06ed62cbe8eca9103c3d111564e6c, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:18,021 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b7b06ed62cbe8eca9103c3d111564e6c, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:54:18,021 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34681] ipc.CallRunner(144): callId: 389 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:39106 deadline: 1681527318019, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b7b06ed62cbe8eca9103c3d111564e6c, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:18,200 INFO [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=704 (bloomFilter=true), to=hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/b7b06ed62cbe8eca9103c3d111564e6c/.tmp/C/c99c8c8578d84adbb4eb965d5c423d59 2023-04-15 02:54:18,204 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/b7b06ed62cbe8eca9103c3d111564e6c/.tmp/A/2b87e81781d04ac4aebf8b72abb33b61 as hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/b7b06ed62cbe8eca9103c3d111564e6c/A/2b87e81781d04ac4aebf8b72abb33b61 2023-04-15 02:54:18,207 INFO [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/b7b06ed62cbe8eca9103c3d111564e6c/A/2b87e81781d04ac4aebf8b72abb33b61, entries=150, sequenceid=704, filesize=25.4 K 2023-04-15 02:54:18,208 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/b7b06ed62cbe8eca9103c3d111564e6c/.tmp/B/cd61e6be1fc947dcaee243656edb94cc as hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/b7b06ed62cbe8eca9103c3d111564e6c/B/cd61e6be1fc947dcaee243656edb94cc 2023-04-15 02:54:18,212 INFO [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/b7b06ed62cbe8eca9103c3d111564e6c/B/cd61e6be1fc947dcaee243656edb94cc, entries=150, sequenceid=704, filesize=11.9 K 2023-04-15 02:54:18,212 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/b7b06ed62cbe8eca9103c3d111564e6c/.tmp/C/c99c8c8578d84adbb4eb965d5c423d59 as hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/b7b06ed62cbe8eca9103c3d111564e6c/C/c99c8c8578d84adbb4eb965d5c423d59 2023-04-15 02:54:18,216 INFO [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/b7b06ed62cbe8eca9103c3d111564e6c/C/c99c8c8578d84adbb4eb965d5c423d59, entries=150, sequenceid=704, filesize=11.9 K 2023-04-15 02:54:18,216 INFO [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~114.05 KB/116790, heapSize ~339.38 KB/347520, currentSize=60.38 KB/61830 for b7b06ed62cbe8eca9103c3d111564e6c in 1257ms, sequenceid=704, compaction requested=true 2023-04-15 02:54:18,216 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for b7b06ed62cbe8eca9103c3d111564e6c: 2023-04-15 02:54:18,217 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681527240182.b7b06ed62cbe8eca9103c3d111564e6c. 2023-04-15 02:54:18,217 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-15 02:54:18,217 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-15 02:54:18,217 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:18,218 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-15 02:54:18,218 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34681,1681527209133' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-15 02:54:18,219 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:18,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:18,219 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:18,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:54:18,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:54:18,220 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:18,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:54:18,220 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-15 02:54:18,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:54:18,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:54:18,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:18,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:54:18,220 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34681,1681527209133' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-15 02:54:18,220 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6e2ff10b[Count = 0] remaining members to acquire global barrier 2023-04-15 02:54:18,221 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-15 02:54:18,221 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:18,221 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:18,221 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:18,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:18,221 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-15 02:54:18,221 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-15 02:54:18,221 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34681,1681527209133' in zk 2023-04-15 02:54:18,221 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:18,221 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-15 02:54:18,222 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-15 02:54:18,222 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:18,222 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-15 02:54:18,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:18,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:54:18,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:54:18,222 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-15 02:54:18,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:54:18,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:54:18,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:54:18,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:18,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:54:18,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:54:18,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:18,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34681,1681527209133': 2023-04-15 02:54:18,235 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34681,1681527209133' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-15 02:54:18,235 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-15 02:54:18,235 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-15 02:54:18,235 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-15 02:54:18,235 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-15 02:54:18,235 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-15 02:54:18,236 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:18,236 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:18,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:18,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:54:18,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:54:18,236 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:18,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:18,236 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-15 02:54:18,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:54:18,237 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-15 02:54:18,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-15 02:54:18,237 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:18,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:54:18,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:18,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:54:18,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:54:18,244 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:18,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:18,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:54:18,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:54:18,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:18,250 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-15 02:54:18,250 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:18,250 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-15 02:54:18,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-15 02:54:18,250 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-15 02:54:18,250 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-15 02:54:18,250 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-15 02:54:18,250 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-15 02:54:18,250 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-15 02:54:18,250 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:54:18,250 DEBUG [Thread-864] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-15 02:54:18,250 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:54:18,250 DEBUG [Thread-864] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-15 02:54:18,251 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:18,251 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:18,251 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:18,251 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-15 02:54:18,251 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:18,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-15 02:54:18,351 DEBUG [Thread-864] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-15 02:54:18,351 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-15 02:54:18,353 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-15 02:54:18,354 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-15 02:54:18,354 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-15 02:54:18,354 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-15 02:54:18,354 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-15 02:54:18,354 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-15 02:54:18,354 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-15 02:54:18,354 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:18,354 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:54:18,355 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:18,355 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-15 02:54:18,355 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-15 02:54:18,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-15 02:54:18,355 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:18,355 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-15 02:54:18,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:54:18,356 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:18,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-15 02:54:18,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:54:18,356 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-15 02:54:18,356 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-15 02:54:18,356 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-15 02:54:18,356 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-15 02:54:18,356 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-15 02:54:18,356 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-15 02:54:18,358 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-15 02:54:18,358 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-15 02:54:18,358 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681527240182.b7b06ed62cbe8eca9103c3d111564e6c. 2023-04-15 02:54:18,358 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681527240182.b7b06ed62cbe8eca9103c3d111564e6c. started... 2023-04-15 02:54:18,358 INFO [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing b7b06ed62cbe8eca9103c3d111564e6c 3/3 column families, dataSize=60.38 KB heapSize=180.05 KB 2023-04-15 02:54:18,358 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK b7b06ed62cbe8eca9103c3d111564e6c, store=A 2023-04-15 02:54:18,358 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-15 02:54:18,358 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK b7b06ed62cbe8eca9103c3d111564e6c, store=B 2023-04-15 02:54:18,358 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-15 02:54:18,358 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK b7b06ed62cbe8eca9103c3d111564e6c, store=C 2023-04-15 02:54:18,358 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-15 02:54:18,387 INFO [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304155e45feaadf0847e5888bb142e4851022 to hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304155e45feaadf0847e5888bb142e4851022 2023-04-15 02:54:18,524 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34681] regionserver.HRegion(9158): Flush requested on b7b06ed62cbe8eca9103c3d111564e6c 2023-04-15 02:54:18,524 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681527240182.b7b06ed62cbe8eca9103c3d111564e6c. as already flushing 2023-04-15 02:54:18,545 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b7b06ed62cbe8eca9103c3d111564e6c, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:54:18,545 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34681] ipc.CallRunner(144): callId: 398 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:39106 deadline: 1681527318543, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b7b06ed62cbe8eca9103c3d111564e6c, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:18,545 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b7b06ed62cbe8eca9103c3d111564e6c, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:54:18,545 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34681] ipc.CallRunner(144): callId: 426 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:39132 deadline: 1681527318544, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b7b06ed62cbe8eca9103c3d111564e6c, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:18,646 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b7b06ed62cbe8eca9103c3d111564e6c, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:54:18,646 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34681] ipc.CallRunner(144): callId: 400 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:39106 deadline: 1681527318646, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b7b06ed62cbe8eca9103c3d111564e6c, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:18,646 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b7b06ed62cbe8eca9103c3d111564e6c, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:54:18,647 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34681] ipc.CallRunner(144): callId: 428 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:39132 deadline: 1681527318646, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b7b06ed62cbe8eca9103c3d111564e6c, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:18,801 INFO [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=716, memsize=20.1 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/b7b06ed62cbe8eca9103c3d111564e6c/.tmp/A/c591c2fdcd9b445bbac205ab6d1faad6 2023-04-15 02:54:18,848 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b7b06ed62cbe8eca9103c3d111564e6c, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:54:18,848 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34681] ipc.CallRunner(144): callId: 402 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:39106 deadline: 1681527318847, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b7b06ed62cbe8eca9103c3d111564e6c, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:18,848 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b7b06ed62cbe8eca9103c3d111564e6c, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:54:18,848 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34681] ipc.CallRunner(144): callId: 430 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:39132 deadline: 1681527318847, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b7b06ed62cbe8eca9103c3d111564e6c, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:19,152 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b7b06ed62cbe8eca9103c3d111564e6c, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:54:19,152 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34681] ipc.CallRunner(144): callId: 432 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:39132 deadline: 1681527319150, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b7b06ed62cbe8eca9103c3d111564e6c, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:19,152 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b7b06ed62cbe8eca9103c3d111564e6c, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:54:19,152 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34681] ipc.CallRunner(144): callId: 404 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:39106 deadline: 1681527319151, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b7b06ed62cbe8eca9103c3d111564e6c, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:19,216 INFO [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=716 (bloomFilter=true), to=hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/b7b06ed62cbe8eca9103c3d111564e6c/.tmp/B/1bb4b536f6f24d5f829bd2d669311137 2023-04-15 02:54:19,626 INFO [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=716 (bloomFilter=true), to=hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/b7b06ed62cbe8eca9103c3d111564e6c/.tmp/C/71abddb901ed4f5bb7ff4e17b9d0cf66 2023-04-15 02:54:19,630 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/b7b06ed62cbe8eca9103c3d111564e6c/.tmp/A/c591c2fdcd9b445bbac205ab6d1faad6 as hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/b7b06ed62cbe8eca9103c3d111564e6c/A/c591c2fdcd9b445bbac205ab6d1faad6 2023-04-15 02:54:19,634 INFO [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/b7b06ed62cbe8eca9103c3d111564e6c/A/c591c2fdcd9b445bbac205ab6d1faad6, entries=150, sequenceid=716, filesize=25.4 K 2023-04-15 02:54:19,635 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/b7b06ed62cbe8eca9103c3d111564e6c/.tmp/B/1bb4b536f6f24d5f829bd2d669311137 as hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/b7b06ed62cbe8eca9103c3d111564e6c/B/1bb4b536f6f24d5f829bd2d669311137 2023-04-15 02:54:19,638 INFO [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/b7b06ed62cbe8eca9103c3d111564e6c/B/1bb4b536f6f24d5f829bd2d669311137, entries=150, sequenceid=716, filesize=11.9 K 2023-04-15 02:54:19,639 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/b7b06ed62cbe8eca9103c3d111564e6c/.tmp/C/71abddb901ed4f5bb7ff4e17b9d0cf66 as hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/b7b06ed62cbe8eca9103c3d111564e6c/C/71abddb901ed4f5bb7ff4e17b9d0cf66 2023-04-15 02:54:19,642 INFO [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/b7b06ed62cbe8eca9103c3d111564e6c/C/71abddb901ed4f5bb7ff4e17b9d0cf66, entries=150, sequenceid=716, filesize=11.9 K 2023-04-15 02:54:19,643 INFO [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~60.38 KB/61830, heapSize ~180 KB/184320, currentSize=114.05 KB/116790 for b7b06ed62cbe8eca9103c3d111564e6c in 1285ms, sequenceid=716, compaction requested=true 2023-04-15 02:54:19,643 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for b7b06ed62cbe8eca9103c3d111564e6c: 2023-04-15 02:54:19,643 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681527240182.b7b06ed62cbe8eca9103c3d111564e6c. 2023-04-15 02:54:19,643 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-15 02:54:19,643 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-15 02:54:19,643 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:19,643 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-15 02:54:19,643 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34681,1681527209133' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-15 02:54:19,644 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:19,644 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:19,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:19,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:54:19,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:54:19,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:54:19,644 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:19,644 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-15 02:54:19,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:54:19,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:54:19,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:19,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:54:19,646 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34681,1681527209133' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-15 02:54:19,646 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5e4fd097[Count = 0] remaining members to acquire global barrier 2023-04-15 02:54:19,646 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-15 02:54:19,646 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:19,646 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:19,646 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:19,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:19,646 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-15 02:54:19,646 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-15 02:54:19,646 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:19,646 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-15 02:54:19,646 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34681,1681527209133' in zk 2023-04-15 02:54:19,647 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-15 02:54:19,647 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:19,647 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-15 02:54:19,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:19,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:54:19,647 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-15 02:54:19,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:54:19,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:54:19,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:54:19,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:54:19,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:19,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:54:19,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:54:19,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:19,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34681,1681527209133': 2023-04-15 02:54:19,649 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34681,1681527209133' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-15 02:54:19,649 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-15 02:54:19,649 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-15 02:54:19,649 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-15 02:54:19,649 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-15 02:54:19,649 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-15 02:54:19,650 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:19,650 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:19,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:19,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:54:19,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:54:19,650 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-15 02:54:19,650 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:19,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:19,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:54:19,651 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-15 02:54:19,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-15 02:54:19,651 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:19,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:54:19,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:19,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:54:19,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:54:19,651 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:19,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:19,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:54:19,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:54:19,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:19,653 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:19,653 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:54:19,653 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-15 02:54:19,653 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:54:19,653 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-15 02:54:19,653 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-15 02:54:19,653 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:19,653 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-15 02:54:19,653 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-15 02:54:19,653 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-15 02:54:19,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-15 02:54:19,653 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-15 02:54:19,653 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:19,653 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:19,653 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:19,653 DEBUG [Thread-864] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-15 02:54:19,653 DEBUG [Thread-864] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-15 02:54:19,653 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-15 02:54:19,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-15 02:54:19,657 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34681] regionserver.HRegion(9158): Flush requested on b7b06ed62cbe8eca9103c3d111564e6c 2023-04-15 02:54:19,657 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing b7b06ed62cbe8eca9103c3d111564e6c 3/3 column families, dataSize=120.76 KB heapSize=360.05 KB 2023-04-15 02:54:19,658 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK b7b06ed62cbe8eca9103c3d111564e6c, store=A 2023-04-15 02:54:19,658 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-15 02:54:19,658 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK b7b06ed62cbe8eca9103c3d111564e6c, store=B 2023-04-15 02:54:19,658 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-15 02:54:19,658 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK b7b06ed62cbe8eca9103c3d111564e6c, store=C 2023-04-15 02:54:19,658 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-15 02:54:19,670 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230415d29b7707d415438fa4fc4885b9851540 to hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230415d29b7707d415438fa4fc4885b9851540 2023-04-15 02:54:19,677 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b7b06ed62cbe8eca9103c3d111564e6c, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:54:19,677 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b7b06ed62cbe8eca9103c3d111564e6c, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:54:19,677 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34681] ipc.CallRunner(144): callId: 411 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:39106 deadline: 1681527319672, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b7b06ed62cbe8eca9103c3d111564e6c, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:19,677 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34681] ipc.CallRunner(144): callId: 439 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:39132 deadline: 1681527319672, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b7b06ed62cbe8eca9103c3d111564e6c, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:19,753 DEBUG [Thread-864] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-15 02:54:19,754 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-15 02:54:19,755 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-15 02:54:19,756 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-15 02:54:19,756 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-15 02:54:19,756 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-15 02:54:19,756 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-15 02:54:19,756 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-15 02:54:19,756 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-15 02:54:19,756 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:19,757 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:54:19,757 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:19,757 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-15 02:54:19,757 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-15 02:54:19,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-15 02:54:19,757 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:19,757 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-15 02:54:19,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:54:19,758 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:19,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-15 02:54:19,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:54:19,758 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-15 02:54:19,758 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-15 02:54:19,758 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-15 02:54:19,758 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-15 02:54:19,758 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-15 02:54:19,758 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-15 02:54:19,758 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-15 02:54:19,758 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681527240182.b7b06ed62cbe8eca9103c3d111564e6c. 2023-04-15 02:54:19,759 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681527240182.b7b06ed62cbe8eca9103c3d111564e6c. started... 2023-04-15 02:54:19,759 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681527240182.b7b06ed62cbe8eca9103c3d111564e6c. as already flushing 2023-04-15 02:54:19,759 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681527240182.b7b06ed62cbe8eca9103c3d111564e6c. 2023-04-15 02:54:19,759 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-15 02:54:19,759 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-15 02:54:19,759 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:19,759 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-15 02:54:19,759 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34681,1681527209133' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-15 02:54:19,760 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:19,760 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:19,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:19,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:54:19,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:54:19,760 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:19,760 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-15 02:54:19,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:54:19,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:54:19,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:54:19,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:19,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:54:19,761 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34681,1681527209133' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-15 02:54:19,761 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@77a4d0ca[Count = 0] remaining members to acquire global barrier 2023-04-15 02:54:19,761 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-15 02:54:19,761 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:19,762 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:19,762 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:19,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:19,762 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-15 02:54:19,762 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-15 02:54:19,762 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34681,1681527209133' in zk 2023-04-15 02:54:19,762 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:19,762 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-15 02:54:19,763 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:19,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:19,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:54:19,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:54:19,763 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-15 02:54:19,763 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-15 02:54:19,763 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-15 02:54:19,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:54:19,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:54:19,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:54:19,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:19,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:54:19,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:54:19,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:19,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34681,1681527209133': 2023-04-15 02:54:19,765 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34681,1681527209133' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-15 02:54:19,765 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-15 02:54:19,765 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-15 02:54:19,765 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-15 02:54:19,765 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-15 02:54:19,765 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-15 02:54:19,766 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:19,766 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:19,766 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-15 02:54:19,766 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:19,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:19,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:19,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:54:19,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:54:19,766 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-15 02:54:19,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-15 02:54:19,766 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:19,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:54:19,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:19,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:54:19,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:54:19,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:54:19,767 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:19,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:19,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:54:19,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:54:19,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:19,769 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:19,769 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-15 02:54:19,769 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-15 02:54:19,769 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-15 02:54:19,769 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:54:19,769 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-15 02:54:19,769 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-15 02:54:19,769 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:54:19,769 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-15 02:54:19,769 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:19,769 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:19,769 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:19,769 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:19,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-15 02:54:19,769 DEBUG [Thread-864] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-15 02:54:19,770 DEBUG [Thread-864] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-15 02:54:19,770 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-15 02:54:19,770 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-15 02:54:19,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-15 02:54:19,778 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b7b06ed62cbe8eca9103c3d111564e6c, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:54:19,778 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34681] ipc.CallRunner(144): callId: 441 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:39132 deadline: 1681527319777, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b7b06ed62cbe8eca9103c3d111564e6c, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:19,779 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b7b06ed62cbe8eca9103c3d111564e6c, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:54:19,779 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34681] ipc.CallRunner(144): callId: 413 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:39106 deadline: 1681527319777, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b7b06ed62cbe8eca9103c3d111564e6c, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:19,870 DEBUG [Thread-864] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-15 02:54:19,870 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-15 02:54:19,871 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-15 02:54:19,872 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-15 02:54:19,872 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-15 02:54:19,872 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-15 02:54:19,872 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-15 02:54:19,872 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-15 02:54:19,872 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-15 02:54:19,873 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:19,873 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:54:19,873 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-15 02:54:19,873 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:19,873 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-15 02:54:19,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-15 02:54:19,874 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:19,874 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-15 02:54:19,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:54:19,874 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:19,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-15 02:54:19,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:54:19,874 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-15 02:54:19,874 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-15 02:54:19,874 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-15 02:54:19,875 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-15 02:54:19,875 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-15 02:54:19,875 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-15 02:54:19,875 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-15 02:54:19,875 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-15 02:54:19,875 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681527240182.b7b06ed62cbe8eca9103c3d111564e6c. 2023-04-15 02:54:19,875 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681527240182.b7b06ed62cbe8eca9103c3d111564e6c. started... 2023-04-15 02:54:19,875 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681527240182.b7b06ed62cbe8eca9103c3d111564e6c. as already flushing 2023-04-15 02:54:19,875 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681527240182.b7b06ed62cbe8eca9103c3d111564e6c. 2023-04-15 02:54:19,875 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-15 02:54:19,875 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-15 02:54:19,875 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:19,875 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-15 02:54:19,875 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34681,1681527209133' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-15 02:54:19,876 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:19,876 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:19,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:19,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:54:19,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:54:19,877 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:19,877 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-15 02:54:19,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:54:19,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:54:19,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:54:19,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:19,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:54:19,878 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34681,1681527209133' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-15 02:54:19,878 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-15 02:54:19,878 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7b9db0e1[Count = 0] remaining members to acquire global barrier 2023-04-15 02:54:19,878 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:19,878 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:19,878 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:19,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:19,878 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-15 02:54:19,878 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-15 02:54:19,878 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34681,1681527209133' in zk 2023-04-15 02:54:19,879 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:19,879 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-15 02:54:19,879 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:19,879 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-15 02:54:19,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:19,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:54:19,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:54:19,879 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-15 02:54:19,879 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-15 02:54:19,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:54:19,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:54:19,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:54:19,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:19,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:54:19,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:54:19,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:19,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34681,1681527209133': 2023-04-15 02:54:19,881 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34681,1681527209133' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-15 02:54:19,881 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-15 02:54:19,881 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-15 02:54:19,881 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-15 02:54:19,881 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-15 02:54:19,881 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-15 02:54:19,882 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:19,882 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:19,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:19,882 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:19,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:54:19,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:54:19,882 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-15 02:54:19,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:19,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:54:19,882 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-15 02:54:19,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-15 02:54:19,883 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:19,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:54:19,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:19,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:54:19,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:54:19,883 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:19,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:19,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:54:19,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:54:19,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:19,885 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:19,885 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:54:19,885 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-15 02:54:19,885 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-15 02:54:19,885 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-15 02:54:19,885 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:54:19,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-15 02:54:19,885 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-15 02:54:19,885 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:19,885 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:19,885 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:19,885 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:19,885 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-15 02:54:19,885 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-15 02:54:19,885 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-15 02:54:19,885 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-15 02:54:19,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-15 02:54:19,886 DEBUG [Thread-864] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-15 02:54:19,886 DEBUG [Thread-864] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-15 02:54:19,980 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b7b06ed62cbe8eca9103c3d111564e6c, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:54:19,980 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34681] ipc.CallRunner(144): callId: 443 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:39132 deadline: 1681527319979, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b7b06ed62cbe8eca9103c3d111564e6c, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:19,980 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b7b06ed62cbe8eca9103c3d111564e6c, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:54:19,980 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34681] ipc.CallRunner(144): callId: 415 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:39106 deadline: 1681527319979, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b7b06ed62cbe8eca9103c3d111564e6c, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:19,986 DEBUG [Thread-864] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-15 02:54:19,986 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-15 02:54:19,987 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-15 02:54:19,988 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-15 02:54:19,988 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-15 02:54:19,988 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-15 02:54:19,988 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-15 02:54:19,988 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-15 02:54:19,988 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-15 02:54:19,989 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:19,989 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:54:19,989 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-15 02:54:19,989 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:19,989 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-15 02:54:19,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-15 02:54:19,990 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:19,990 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-15 02:54:19,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:54:19,990 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:19,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-15 02:54:19,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:54:19,990 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-15 02:54:19,991 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-15 02:54:19,991 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-15 02:54:19,991 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-15 02:54:19,991 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-15 02:54:19,991 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-15 02:54:19,991 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-15 02:54:19,991 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-15 02:54:19,991 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681527240182.b7b06ed62cbe8eca9103c3d111564e6c. 2023-04-15 02:54:19,991 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681527240182.b7b06ed62cbe8eca9103c3d111564e6c. started... 2023-04-15 02:54:19,991 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681527240182.b7b06ed62cbe8eca9103c3d111564e6c. as already flushing 2023-04-15 02:54:19,991 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681527240182.b7b06ed62cbe8eca9103c3d111564e6c. 2023-04-15 02:54:19,991 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-15 02:54:19,991 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-15 02:54:19,991 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:19,991 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-15 02:54:19,991 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34681,1681527209133' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-15 02:54:19,992 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:19,992 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:19,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:19,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:54:19,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:54:19,993 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:19,993 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-15 02:54:19,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:54:19,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:54:19,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:54:19,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:19,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:54:19,994 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34681,1681527209133' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-15 02:54:19,994 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6aabb130[Count = 0] remaining members to acquire global barrier 2023-04-15 02:54:19,994 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-15 02:54:19,994 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:19,994 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:19,994 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:19,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:19,994 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-15 02:54:19,994 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-15 02:54:19,994 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34681,1681527209133' in zk 2023-04-15 02:54:19,994 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:19,994 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-15 02:54:19,995 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:19,995 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-15 02:54:19,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:19,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:54:19,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:54:19,995 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-15 02:54:19,995 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-15 02:54:19,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:54:19,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:54:19,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:54:19,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:19,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:54:19,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:54:19,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:19,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34681,1681527209133': 2023-04-15 02:54:19,997 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34681,1681527209133' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-15 02:54:19,997 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-15 02:54:19,997 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-15 02:54:19,997 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-15 02:54:19,997 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-15 02:54:19,997 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-15 02:54:19,998 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:19,998 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:19,998 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:19,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:19,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:54:19,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:54:19,998 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-15 02:54:19,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:19,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:54:19,998 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:19,998 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-15 02:54:19,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-15 02:54:19,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:54:19,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:54:19,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:19,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:54:19,999 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:19,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:20,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:54:20,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:54:20,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:20,001 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:20,001 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-15 02:54:20,001 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-15 02:54:20,001 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-15 02:54:20,001 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-15 02:54:20,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-15 02:54:20,001 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:54:20,001 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:54:20,001 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:20,001 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:20,001 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:20,001 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:20,001 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-15 02:54:20,001 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-15 02:54:20,001 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-15 02:54:20,001 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-15 02:54:20,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-15 02:54:20,002 DEBUG [Thread-864] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-15 02:54:20,002 DEBUG [Thread-864] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-15 02:54:20,074 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=737, memsize=40.3 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/b7b06ed62cbe8eca9103c3d111564e6c/.tmp/A/efc42ad066a04d8da406d201b8de5f21 2023-04-15 02:54:20,102 DEBUG [Thread-864] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-15 02:54:20,103 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-15 02:54:20,104 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-15 02:54:20,105 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-15 02:54:20,105 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-15 02:54:20,105 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-15 02:54:20,105 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-15 02:54:20,105 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-15 02:54:20,105 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-15 02:54:20,105 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:20,105 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:54:20,106 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-15 02:54:20,106 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:20,106 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-15 02:54:20,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-15 02:54:20,106 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:20,106 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-15 02:54:20,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:54:20,107 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:20,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-15 02:54:20,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:54:20,107 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-15 02:54:20,107 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-15 02:54:20,107 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-15 02:54:20,107 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-15 02:54:20,107 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-15 02:54:20,107 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-15 02:54:20,107 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-15 02:54:20,107 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-15 02:54:20,107 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681527240182.b7b06ed62cbe8eca9103c3d111564e6c. 2023-04-15 02:54:20,108 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681527240182.b7b06ed62cbe8eca9103c3d111564e6c. started... 2023-04-15 02:54:20,108 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681527240182.b7b06ed62cbe8eca9103c3d111564e6c. as already flushing 2023-04-15 02:54:20,108 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681527240182.b7b06ed62cbe8eca9103c3d111564e6c. 2023-04-15 02:54:20,108 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-15 02:54:20,108 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-15 02:54:20,108 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:20,108 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-15 02:54:20,108 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34681,1681527209133' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-15 02:54:20,109 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:20,109 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:20,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:20,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:54:20,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:54:20,109 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:20,109 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-15 02:54:20,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:54:20,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:54:20,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:54:20,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:20,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:54:20,111 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34681,1681527209133' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-15 02:54:20,111 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4a400f9a[Count = 0] remaining members to acquire global barrier 2023-04-15 02:54:20,111 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-15 02:54:20,112 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:20,112 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:20,112 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:20,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:20,112 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-15 02:54:20,112 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-15 02:54:20,112 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34681,1681527209133' in zk 2023-04-15 02:54:20,112 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:20,112 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-15 02:54:20,113 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:20,113 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-15 02:54:20,113 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-15 02:54:20,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:20,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:54:20,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:54:20,113 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-15 02:54:20,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:54:20,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:54:20,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:54:20,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:20,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:54:20,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:54:20,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:20,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34681,1681527209133': 2023-04-15 02:54:20,115 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34681,1681527209133' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-15 02:54:20,115 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-15 02:54:20,115 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-15 02:54:20,115 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-15 02:54:20,115 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-15 02:54:20,115 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-15 02:54:20,116 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:20,116 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:20,116 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:20,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:20,116 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-15 02:54:20,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:20,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:54:20,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:54:20,116 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-15 02:54:20,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-15 02:54:20,117 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:20,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:54:20,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:20,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:54:20,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:54:20,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:54:20,118 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:20,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:20,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:54:20,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:54:20,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:20,120 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:20,120 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:54:20,120 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-15 02:54:20,120 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:54:20,120 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-15 02:54:20,120 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-15 02:54:20,120 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-15 02:54:20,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-15 02:54:20,120 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-15 02:54:20,120 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-15 02:54:20,120 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-15 02:54:20,120 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:20,120 DEBUG [Thread-864] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-15 02:54:20,120 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:20,120 DEBUG [Thread-864] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-15 02:54:20,121 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:20,121 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:20,121 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-15 02:54:20,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-15 02:54:20,221 DEBUG [Thread-864] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-15 02:54:20,221 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-15 02:54:20,223 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-15 02:54:20,224 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-15 02:54:20,224 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-15 02:54:20,224 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-15 02:54:20,224 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-15 02:54:20,224 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-15 02:54:20,224 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-15 02:54:20,224 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:20,224 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:54:20,225 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-15 02:54:20,225 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:20,225 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-15 02:54:20,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-15 02:54:20,225 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:20,225 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-15 02:54:20,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:54:20,225 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:20,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-15 02:54:20,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:54:20,226 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-15 02:54:20,226 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-15 02:54:20,226 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-15 02:54:20,226 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-15 02:54:20,226 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-15 02:54:20,226 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-15 02:54:20,226 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-15 02:54:20,226 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-15 02:54:20,226 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681527240182.b7b06ed62cbe8eca9103c3d111564e6c. 2023-04-15 02:54:20,226 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681527240182.b7b06ed62cbe8eca9103c3d111564e6c. started... 2023-04-15 02:54:20,226 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681527240182.b7b06ed62cbe8eca9103c3d111564e6c. as already flushing 2023-04-15 02:54:20,226 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681527240182.b7b06ed62cbe8eca9103c3d111564e6c. 2023-04-15 02:54:20,226 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-15 02:54:20,226 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-15 02:54:20,226 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:20,226 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-15 02:54:20,226 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34681,1681527209133' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-15 02:54:20,227 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:20,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:20,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:54:20,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:54:20,227 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:20,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:54:20,228 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:20,228 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-15 02:54:20,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:54:20,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:54:20,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:20,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:54:20,228 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34681,1681527209133' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-15 02:54:20,228 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-15 02:54:20,228 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1bb3b498[Count = 0] remaining members to acquire global barrier 2023-04-15 02:54:20,228 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:20,229 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:20,229 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:20,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:20,229 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-15 02:54:20,229 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-15 02:54:20,229 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34681,1681527209133' in zk 2023-04-15 02:54:20,229 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:20,229 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-15 02:54:20,230 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:20,230 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-15 02:54:20,230 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-15 02:54:20,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:20,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:54:20,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:54:20,230 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-15 02:54:20,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:54:20,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:54:20,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:54:20,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:20,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:54:20,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:54:20,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:20,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34681,1681527209133': 2023-04-15 02:54:20,232 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34681,1681527209133' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-15 02:54:20,232 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-15 02:54:20,232 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-15 02:54:20,232 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-15 02:54:20,232 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-15 02:54:20,232 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-15 02:54:20,233 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:20,233 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:20,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:20,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:54:20,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:54:20,233 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:20,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:20,233 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-15 02:54:20,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:54:20,233 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:20,233 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-15 02:54:20,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-15 02:54:20,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:54:20,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:20,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:54:20,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:54:20,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:20,234 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:20,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:54:20,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:54:20,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:20,235 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:20,235 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-15 02:54:20,235 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:54:20,235 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-15 02:54:20,235 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-15 02:54:20,235 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:54:20,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-15 02:54:20,235 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-15 02:54:20,235 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-15 02:54:20,235 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-15 02:54:20,235 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:20,236 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:20,236 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:20,236 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:20,236 DEBUG [Thread-864] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-15 02:54:20,236 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-15 02:54:20,236 DEBUG [Thread-864] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-15 02:54:20,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-15 02:54:20,284 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b7b06ed62cbe8eca9103c3d111564e6c, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:54:20,284 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34681] ipc.CallRunner(144): callId: 445 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:39132 deadline: 1681527320282, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b7b06ed62cbe8eca9103c3d111564e6c, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:20,285 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b7b06ed62cbe8eca9103c3d111564e6c, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:54:20,285 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34681] ipc.CallRunner(144): callId: 417 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:39106 deadline: 1681527320283, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b7b06ed62cbe8eca9103c3d111564e6c, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:20,336 DEBUG [Thread-864] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-15 02:54:20,336 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-15 02:54:20,338 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-15 02:54:20,338 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-15 02:54:20,338 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-15 02:54:20,338 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-15 02:54:20,338 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-15 02:54:20,338 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-15 02:54:20,339 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-15 02:54:20,339 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:20,339 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:54:20,340 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:20,340 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-15 02:54:20,341 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-15 02:54:20,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-15 02:54:20,341 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:20,341 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-15 02:54:20,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:54:20,341 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:20,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-15 02:54:20,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:54:20,341 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-15 02:54:20,342 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-15 02:54:20,342 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-15 02:54:20,342 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-15 02:54:20,342 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-15 02:54:20,342 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-15 02:54:20,342 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-15 02:54:20,342 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-15 02:54:20,342 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681527240182.b7b06ed62cbe8eca9103c3d111564e6c. 2023-04-15 02:54:20,342 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681527240182.b7b06ed62cbe8eca9103c3d111564e6c. started... 2023-04-15 02:54:20,342 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681527240182.b7b06ed62cbe8eca9103c3d111564e6c. as already flushing 2023-04-15 02:54:20,342 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681527240182.b7b06ed62cbe8eca9103c3d111564e6c. 2023-04-15 02:54:20,342 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-15 02:54:20,342 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-15 02:54:20,342 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:20,343 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-15 02:54:20,343 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34681,1681527209133' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-15 02:54:20,343 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:20,343 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:20,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:20,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:54:20,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:54:20,344 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:20,344 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-15 02:54:20,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:54:20,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:54:20,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:54:20,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:20,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:54:20,345 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34681,1681527209133' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-15 02:54:20,345 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@d3f15b7[Count = 0] remaining members to acquire global barrier 2023-04-15 02:54:20,345 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-15 02:54:20,345 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:20,345 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:20,345 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:20,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:20,345 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-15 02:54:20,345 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:20,345 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-15 02:54:20,345 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-15 02:54:20,345 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34681,1681527209133' in zk 2023-04-15 02:54:20,346 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:20,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:20,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:54:20,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:54:20,346 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-15 02:54:20,346 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-15 02:54:20,346 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-15 02:54:20,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:54:20,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:54:20,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:54:20,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:20,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:54:20,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:54:20,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:20,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34681,1681527209133': 2023-04-15 02:54:20,348 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34681,1681527209133' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-15 02:54:20,348 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-15 02:54:20,348 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-15 02:54:20,348 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-15 02:54:20,348 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-15 02:54:20,348 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-15 02:54:20,349 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:20,349 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:20,349 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-15 02:54:20,349 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:20,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:20,349 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-15 02:54:20,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-15 02:54:20,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:20,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:20,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:54:20,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:54:20,350 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:20,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:54:20,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:54:20,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:54:20,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:54:20,351 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:20,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:20,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:54:20,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:54:20,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:20,352 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:20,352 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:54:20,352 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-15 02:54:20,352 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:54:20,352 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:20,352 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:20,352 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:20,352 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:20,352 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-15 02:54:20,352 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-15 02:54:20,352 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-15 02:54:20,352 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-15 02:54:20,352 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-15 02:54:20,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-15 02:54:20,353 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-15 02:54:20,353 DEBUG [Thread-864] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-15 02:54:20,353 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-15 02:54:20,353 DEBUG [Thread-864] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-15 02:54:20,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-15 02:54:20,453 DEBUG [Thread-864] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-15 02:54:20,454 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-15 02:54:20,455 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-15 02:54:20,456 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-15 02:54:20,456 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-15 02:54:20,456 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-15 02:54:20,456 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-15 02:54:20,456 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-15 02:54:20,456 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-15 02:54:20,456 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:20,456 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:54:20,457 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-15 02:54:20,457 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:20,457 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-15 02:54:20,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-15 02:54:20,457 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:20,457 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-15 02:54:20,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:54:20,457 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:20,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-15 02:54:20,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:54:20,458 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-15 02:54:20,458 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-15 02:54:20,458 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-15 02:54:20,458 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-15 02:54:20,458 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-15 02:54:20,458 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-15 02:54:20,458 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-15 02:54:20,458 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-15 02:54:20,458 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681527240182.b7b06ed62cbe8eca9103c3d111564e6c. 2023-04-15 02:54:20,458 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681527240182.b7b06ed62cbe8eca9103c3d111564e6c. started... 2023-04-15 02:54:20,458 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681527240182.b7b06ed62cbe8eca9103c3d111564e6c. as already flushing 2023-04-15 02:54:20,458 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681527240182.b7b06ed62cbe8eca9103c3d111564e6c. 2023-04-15 02:54:20,458 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-15 02:54:20,458 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-15 02:54:20,458 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:20,458 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-15 02:54:20,458 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34681,1681527209133' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-15 02:54:20,459 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:20,459 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:20,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:20,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:54:20,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:54:20,459 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:20,459 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-15 02:54:20,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:54:20,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:54:20,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:54:20,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:20,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:54:20,460 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34681,1681527209133' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-15 02:54:20,460 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3aaa1711[Count = 0] remaining members to acquire global barrier 2023-04-15 02:54:20,460 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-15 02:54:20,460 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:20,461 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:20,461 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:20,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:20,461 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-15 02:54:20,461 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-15 02:54:20,461 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34681,1681527209133' in zk 2023-04-15 02:54:20,461 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:20,461 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-15 02:54:20,462 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-15 02:54:20,462 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:20,462 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-15 02:54:20,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:20,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:54:20,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:54:20,462 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-15 02:54:20,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:54:20,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:54:20,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:54:20,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:20,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:54:20,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:54:20,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:20,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34681,1681527209133': 2023-04-15 02:54:20,463 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34681,1681527209133' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-15 02:54:20,463 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-15 02:54:20,463 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-15 02:54:20,463 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-15 02:54:20,463 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-15 02:54:20,463 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-15 02:54:20,464 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:20,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:20,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:54:20,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:54:20,464 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:20,464 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-15 02:54:20,464 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:20,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:20,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:54:20,465 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:20,465 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-15 02:54:20,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-15 02:54:20,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:54:20,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:20,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:54:20,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:54:20,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:20,466 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:20,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:54:20,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:54:20,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:20,467 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:20,467 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-15 02:54:20,467 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-15 02:54:20,467 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-15 02:54:20,467 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-15 02:54:20,467 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-15 02:54:20,467 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:54:20,467 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-15 02:54:20,467 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-15 02:54:20,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-15 02:54:20,467 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:54:20,467 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:20,467 DEBUG [Thread-864] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-15 02:54:20,468 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:20,468 DEBUG [Thread-864] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-15 02:54:20,468 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-15 02:54:20,468 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:20,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-15 02:54:20,468 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:20,485 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=40.25 KB at sequenceid=737 (bloomFilter=true), to=hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/b7b06ed62cbe8eca9103c3d111564e6c/.tmp/B/c87760f24be14299b598da8c0f5ec6f5 2023-04-15 02:54:20,568 DEBUG [Thread-864] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-15 02:54:20,568 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-15 02:54:20,569 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-15 02:54:20,570 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-15 02:54:20,570 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-15 02:54:20,570 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-15 02:54:20,570 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-15 02:54:20,570 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-15 02:54:20,570 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-15 02:54:20,571 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:20,571 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:54:20,572 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-15 02:54:20,572 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:20,572 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-15 02:54:20,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-15 02:54:20,573 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:20,573 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-15 02:54:20,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:54:20,573 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:20,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-15 02:54:20,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:54:20,573 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-15 02:54:20,573 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-15 02:54:20,573 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-15 02:54:20,573 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-15 02:54:20,574 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-15 02:54:20,574 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-15 02:54:20,574 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-15 02:54:20,574 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-15 02:54:20,574 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681527240182.b7b06ed62cbe8eca9103c3d111564e6c. 2023-04-15 02:54:20,574 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681527240182.b7b06ed62cbe8eca9103c3d111564e6c. started... 2023-04-15 02:54:20,574 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681527240182.b7b06ed62cbe8eca9103c3d111564e6c. as already flushing 2023-04-15 02:54:20,574 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681527240182.b7b06ed62cbe8eca9103c3d111564e6c. 2023-04-15 02:54:20,574 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-15 02:54:20,574 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-15 02:54:20,574 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:20,574 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-15 02:54:20,574 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34681,1681527209133' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-15 02:54:20,575 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:20,575 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:20,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:20,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:54:20,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:54:20,575 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:20,575 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-15 02:54:20,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:54:20,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:54:20,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:54:20,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:20,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:54:20,576 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34681,1681527209133' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-15 02:54:20,576 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@705cf301[Count = 0] remaining members to acquire global barrier 2023-04-15 02:54:20,576 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-15 02:54:20,576 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:20,577 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:20,577 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:20,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:20,577 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-15 02:54:20,577 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-15 02:54:20,577 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:20,577 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-15 02:54:20,577 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34681,1681527209133' in zk 2023-04-15 02:54:20,577 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:20,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:20,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:54:20,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:54:20,578 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-15 02:54:20,578 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-15 02:54:20,578 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-15 02:54:20,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:54:20,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:54:20,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:54:20,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:20,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:54:20,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:54:20,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:20,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34681,1681527209133': 2023-04-15 02:54:20,579 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34681,1681527209133' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-15 02:54:20,579 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-15 02:54:20,579 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-15 02:54:20,579 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-15 02:54:20,579 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-15 02:54:20,579 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-15 02:54:20,580 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:20,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:20,580 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:20,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:54:20,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:54:20,580 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:20,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:20,580 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-15 02:54:20,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:54:20,580 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-15 02:54:20,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-15 02:54:20,581 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:20,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:54:20,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:20,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:54:20,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:54:20,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:20,581 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:20,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:54:20,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:54:20,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:20,583 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:20,583 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:54:20,583 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-15 02:54:20,583 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-15 02:54:20,583 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-15 02:54:20,583 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:54:20,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-15 02:54:20,583 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-15 02:54:20,583 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-15 02:54:20,583 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-15 02:54:20,583 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-15 02:54:20,583 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:20,583 DEBUG [Thread-864] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-15 02:54:20,583 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:20,583 DEBUG [Thread-864] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-15 02:54:20,583 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-15 02:54:20,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-15 02:54:20,583 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:20,584 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:20,684 DEBUG [Thread-864] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-15 02:54:20,684 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-15 02:54:20,685 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-15 02:54:20,686 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-15 02:54:20,686 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-15 02:54:20,686 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-15 02:54:20,686 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-15 02:54:20,686 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-15 02:54:20,686 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-15 02:54:20,686 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:20,686 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:54:20,687 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:20,687 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-15 02:54:20,687 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-15 02:54:20,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-15 02:54:20,687 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:20,687 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-15 02:54:20,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:54:20,688 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:20,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-15 02:54:20,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:54:20,688 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-15 02:54:20,688 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-15 02:54:20,688 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-15 02:54:20,688 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-15 02:54:20,688 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-15 02:54:20,688 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-15 02:54:20,688 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-15 02:54:20,688 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-15 02:54:20,688 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681527240182.b7b06ed62cbe8eca9103c3d111564e6c. 2023-04-15 02:54:20,689 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681527240182.b7b06ed62cbe8eca9103c3d111564e6c. started... 2023-04-15 02:54:20,689 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681527240182.b7b06ed62cbe8eca9103c3d111564e6c. as already flushing 2023-04-15 02:54:20,689 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681527240182.b7b06ed62cbe8eca9103c3d111564e6c. 2023-04-15 02:54:20,689 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-15 02:54:20,689 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-15 02:54:20,689 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:20,689 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-15 02:54:20,689 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34681,1681527209133' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-15 02:54:20,690 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:20,690 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:20,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:20,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:54:20,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:54:20,690 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:20,690 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-15 02:54:20,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:54:20,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:54:20,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:54:20,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:20,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:54:20,691 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34681,1681527209133' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-15 02:54:20,691 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-15 02:54:20,691 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@76cd8ee6[Count = 0] remaining members to acquire global barrier 2023-04-15 02:54:20,691 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:20,691 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:20,691 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:20,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:20,691 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-15 02:54:20,691 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:20,691 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-15 02:54:20,691 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-15 02:54:20,692 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34681,1681527209133' in zk 2023-04-15 02:54:20,692 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:20,692 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-15 02:54:20,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:20,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:54:20,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:54:20,692 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-15 02:54:20,692 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-15 02:54:20,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:54:20,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:54:20,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:54:20,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:20,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:54:20,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:54:20,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:20,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34681,1681527209133': 2023-04-15 02:54:20,694 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34681,1681527209133' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-15 02:54:20,694 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-15 02:54:20,694 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-15 02:54:20,694 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-15 02:54:20,694 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-15 02:54:20,694 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-15 02:54:20,694 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:20,694 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:20,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:20,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:54:20,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:54:20,694 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:20,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:20,694 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-15 02:54:20,695 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-15 02:54:20,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-15 02:54:20,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:54:20,695 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:20,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:20,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:54:20,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:54:20,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:54:20,696 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:20,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:20,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:54:20,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:54:20,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:20,697 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:20,697 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-15 02:54:20,697 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:54:20,697 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-15 02:54:20,697 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-15 02:54:20,697 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-15 02:54:20,697 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:54:20,697 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-15 02:54:20,697 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-15 02:54:20,697 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-15 02:54:20,697 DEBUG [Thread-864] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-15 02:54:20,697 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:20,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-15 02:54:20,697 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:20,697 DEBUG [Thread-864] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-15 02:54:20,697 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:20,697 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:20,698 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-15 02:54:20,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-15 02:54:20,787 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b7b06ed62cbe8eca9103c3d111564e6c, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:54:20,787 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34681] ipc.CallRunner(144): callId: 447 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:39132 deadline: 1681527320785, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b7b06ed62cbe8eca9103c3d111564e6c, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:20,791 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b7b06ed62cbe8eca9103c3d111564e6c, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:54:20,791 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34681] ipc.CallRunner(144): callId: 419 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:39106 deadline: 1681527320790, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b7b06ed62cbe8eca9103c3d111564e6c, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:20,797 DEBUG [Thread-864] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-15 02:54:20,798 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-15 02:54:20,799 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-15 02:54:20,800 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-15 02:54:20,800 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-15 02:54:20,800 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-15 02:54:20,800 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-15 02:54:20,800 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-15 02:54:20,800 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-15 02:54:20,801 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:20,801 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:54:20,801 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-15 02:54:20,801 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:20,801 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-15 02:54:20,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-15 02:54:20,801 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:20,801 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-15 02:54:20,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:54:20,802 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:20,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-15 02:54:20,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:54:20,802 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-15 02:54:20,802 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-15 02:54:20,802 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-15 02:54:20,803 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-15 02:54:20,803 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-15 02:54:20,803 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-15 02:54:20,803 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-15 02:54:20,803 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-15 02:54:20,803 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681527240182.b7b06ed62cbe8eca9103c3d111564e6c. 2023-04-15 02:54:20,803 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681527240182.b7b06ed62cbe8eca9103c3d111564e6c. started... 2023-04-15 02:54:20,803 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681527240182.b7b06ed62cbe8eca9103c3d111564e6c. as already flushing 2023-04-15 02:54:20,803 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681527240182.b7b06ed62cbe8eca9103c3d111564e6c. 2023-04-15 02:54:20,803 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-15 02:54:20,803 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-15 02:54:20,803 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:20,803 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-15 02:54:20,803 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34681,1681527209133' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-15 02:54:20,804 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:20,804 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:20,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:20,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:54:20,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:54:20,804 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:20,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:54:20,804 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-15 02:54:20,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:54:20,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:54:20,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:20,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:54:20,805 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34681,1681527209133' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-15 02:54:20,805 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-15 02:54:20,805 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@32abf25[Count = 0] remaining members to acquire global barrier 2023-04-15 02:54:20,805 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:20,806 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:20,806 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:20,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:20,806 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-15 02:54:20,806 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-15 02:54:20,806 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:20,806 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-15 02:54:20,806 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34681,1681527209133' in zk 2023-04-15 02:54:20,806 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:20,806 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-15 02:54:20,806 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-15 02:54:20,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:20,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:54:20,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:54:20,806 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-15 02:54:20,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:54:20,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:54:20,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:54:20,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:20,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:54:20,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:54:20,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:20,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34681,1681527209133': 2023-04-15 02:54:20,809 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34681,1681527209133' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-15 02:54:20,809 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-15 02:54:20,809 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-15 02:54:20,809 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-15 02:54:20,809 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-15 02:54:20,809 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-15 02:54:20,810 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:20,810 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-15 02:54:20,810 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:20,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:20,810 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:20,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:20,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:54:20,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:54:20,810 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-15 02:54:20,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-15 02:54:20,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:54:20,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:20,811 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:20,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:54:20,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:54:20,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:54:20,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:20,812 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:20,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:54:20,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:54:20,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:20,813 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:20,813 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:54:20,813 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:54:20,813 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:20,813 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-15 02:54:20,813 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-15 02:54:20,813 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-15 02:54:20,813 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:20,813 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-15 02:54:20,813 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:20,813 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-15 02:54:20,813 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-15 02:54:20,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-15 02:54:20,813 DEBUG [Thread-864] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-15 02:54:20,813 DEBUG [Thread-864] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-15 02:54:20,813 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:20,814 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-15 02:54:20,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-15 02:54:20,894 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=40.25 KB at sequenceid=737 (bloomFilter=true), to=hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/b7b06ed62cbe8eca9103c3d111564e6c/.tmp/C/65fa15372bed4cf0b6d1ef5b6d16d9cd 2023-04-15 02:54:20,898 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/b7b06ed62cbe8eca9103c3d111564e6c/.tmp/A/efc42ad066a04d8da406d201b8de5f21 as hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/b7b06ed62cbe8eca9103c3d111564e6c/A/efc42ad066a04d8da406d201b8de5f21 2023-04-15 02:54:20,901 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/b7b06ed62cbe8eca9103c3d111564e6c/A/efc42ad066a04d8da406d201b8de5f21, entries=200, sequenceid=737, filesize=32.3 K 2023-04-15 02:54:20,902 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/b7b06ed62cbe8eca9103c3d111564e6c/.tmp/B/c87760f24be14299b598da8c0f5ec6f5 as hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/b7b06ed62cbe8eca9103c3d111564e6c/B/c87760f24be14299b598da8c0f5ec6f5 2023-04-15 02:54:20,905 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/b7b06ed62cbe8eca9103c3d111564e6c/B/c87760f24be14299b598da8c0f5ec6f5, entries=150, sequenceid=737, filesize=11.9 K 2023-04-15 02:54:20,906 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/b7b06ed62cbe8eca9103c3d111564e6c/.tmp/C/65fa15372bed4cf0b6d1ef5b6d16d9cd as hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/b7b06ed62cbe8eca9103c3d111564e6c/C/65fa15372bed4cf0b6d1ef5b6d16d9cd 2023-04-15 02:54:20,911 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/b7b06ed62cbe8eca9103c3d111564e6c/C/65fa15372bed4cf0b6d1ef5b6d16d9cd, entries=150, sequenceid=737, filesize=11.9 K 2023-04-15 02:54:20,912 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~120.76 KB/123660, heapSize ~360 KB/368640, currentSize=60.38 KB/61830 for b7b06ed62cbe8eca9103c3d111564e6c in 1255ms, sequenceid=737, compaction requested=true 2023-04-15 02:54:20,912 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for b7b06ed62cbe8eca9103c3d111564e6c: 2023-04-15 02:54:20,912 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-15 02:54:20,913 DEBUG [RS:0;jenkins-hbase20:34681-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-04-15 02:54:20,913 DEBUG [RS:0;jenkins-hbase20:34681-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-04-15 02:54:20,913 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-15 02:54:20,913 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-15 02:54:20,913 DEBUG [Thread-864] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-15 02:54:20,914 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-15 02:54:20,915 DEBUG [RS:0;jenkins-hbase20:34681-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 137782 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-04-15 02:54:20,915 DEBUG [RS:0;jenkins-hbase20:34681-shortCompactions-0] regionserver.HStore(1912): b7b06ed62cbe8eca9103c3d111564e6c/A is initiating minor compaction (all files) 2023-04-15 02:54:20,915 DEBUG [RS:0;jenkins-hbase20:34681-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 61442 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-04-15 02:54:20,915 INFO [RS:0;jenkins-hbase20:34681-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of b7b06ed62cbe8eca9103c3d111564e6c/A in TestAcidGuarantees,,1681527240182.b7b06ed62cbe8eca9103c3d111564e6c. 2023-04-15 02:54:20,915 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-15 02:54:20,916 INFO [RS:0;jenkins-hbase20:34681-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/b7b06ed62cbe8eca9103c3d111564e6c/A/e8204d9538ec45a184fb961fe5527933, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/b7b06ed62cbe8eca9103c3d111564e6c/A/dcda3bbd8a00469a96653f91fa83ddad, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/b7b06ed62cbe8eca9103c3d111564e6c/A/2b87e81781d04ac4aebf8b72abb33b61, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/b7b06ed62cbe8eca9103c3d111564e6c/A/c591c2fdcd9b445bbac205ab6d1faad6, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/b7b06ed62cbe8eca9103c3d111564e6c/A/efc42ad066a04d8da406d201b8de5f21] into tmpdir=hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/b7b06ed62cbe8eca9103c3d111564e6c/.tmp, totalSize=134.6 K 2023-04-15 02:54:20,915 DEBUG [RS:0;jenkins-hbase20:34681-longCompactions-0] regionserver.HStore(1912): b7b06ed62cbe8eca9103c3d111564e6c/B is initiating minor compaction (all files) 2023-04-15 02:54:20,916 INFO [RS:0;jenkins-hbase20:34681-longCompactions-0] regionserver.HRegion(2259): Starting compaction of b7b06ed62cbe8eca9103c3d111564e6c/B in TestAcidGuarantees,,1681527240182.b7b06ed62cbe8eca9103c3d111564e6c. 2023-04-15 02:54:20,916 INFO [RS:0;jenkins-hbase20:34681-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/b7b06ed62cbe8eca9103c3d111564e6c/B/9168c283c4e345d9b692f4ad877325c0, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/b7b06ed62cbe8eca9103c3d111564e6c/B/51687ca44ed04e20b7384f9f2e53d0da, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/b7b06ed62cbe8eca9103c3d111564e6c/B/cd61e6be1fc947dcaee243656edb94cc, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/b7b06ed62cbe8eca9103c3d111564e6c/B/1bb4b536f6f24d5f829bd2d669311137, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/b7b06ed62cbe8eca9103c3d111564e6c/B/c87760f24be14299b598da8c0f5ec6f5] into tmpdir=hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/b7b06ed62cbe8eca9103c3d111564e6c/.tmp, totalSize=60.0 K 2023-04-15 02:54:20,916 DEBUG [RS:0;jenkins-hbase20:34681-shortCompactions-0] compactions.Compactor(207): Compacting e8204d9538ec45a184fb961fe5527933, keycount=150, bloomtype=ROW, size=25.9 K, encoding=NONE, compression=NONE, seqNum=669, earliestPutTs=1681527254073 2023-04-15 02:54:20,916 DEBUG [RS:0;jenkins-hbase20:34681-longCompactions-0] compactions.Compactor(207): Compacting 9168c283c4e345d9b692f4ad877325c0, keycount=150, bloomtype=ROW, size=12.4 K, encoding=NONE, compression=NONE, seqNum=669, earliestPutTs=1681527254073 2023-04-15 02:54:20,916 DEBUG [RS:0;jenkins-hbase20:34681-shortCompactions-0] compactions.Compactor(207): Compacting dcda3bbd8a00469a96653f91fa83ddad, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=681, earliestPutTs=1681527254092 2023-04-15 02:54:20,917 DEBUG [RS:0;jenkins-hbase20:34681-longCompactions-0] compactions.Compactor(207): Compacting 51687ca44ed04e20b7384f9f2e53d0da, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=681, earliestPutTs=1681527254092 2023-04-15 02:54:20,917 DEBUG [RS:0;jenkins-hbase20:34681-shortCompactions-0] compactions.Compactor(207): Compacting 2b87e81781d04ac4aebf8b72abb33b61, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=704, earliestPutTs=1681527255247 2023-04-15 02:54:20,917 DEBUG [RS:0;jenkins-hbase20:34681-longCompactions-0] compactions.Compactor(207): Compacting cd61e6be1fc947dcaee243656edb94cc, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=704, earliestPutTs=1681527255247 2023-04-15 02:54:20,917 DEBUG [RS:0;jenkins-hbase20:34681-shortCompactions-0] compactions.Compactor(207): Compacting c591c2fdcd9b445bbac205ab6d1faad6, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=716, earliestPutTs=1681527257392 2023-04-15 02:54:20,917 DEBUG [RS:0;jenkins-hbase20:34681-longCompactions-0] compactions.Compactor(207): Compacting 1bb4b536f6f24d5f829bd2d669311137, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=716, earliestPutTs=1681527257392 2023-04-15 02:54:20,917 DEBUG [RS:0;jenkins-hbase20:34681-shortCompactions-0] compactions.Compactor(207): Compacting efc42ad066a04d8da406d201b8de5f21, keycount=200, bloomtype=ROW, size=32.3 K, encoding=NONE, compression=NONE, seqNum=737, earliestPutTs=1681527258539 2023-04-15 02:54:20,918 DEBUG [RS:0;jenkins-hbase20:34681-longCompactions-0] compactions.Compactor(207): Compacting c87760f24be14299b598da8c0f5ec6f5, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=737, earliestPutTs=1681527258539 2023-04-15 02:54:20,921 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-15 02:54:20,922 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-15 02:54:20,922 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-15 02:54:20,922 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-15 02:54:20,922 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-15 02:54:20,922 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-15 02:54:20,922 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:20,922 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:54:20,923 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-15 02:54:20,923 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:20,923 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-15 02:54:20,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-15 02:54:20,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:54:20,923 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:20,923 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:20,924 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-15 02:54:20,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-15 02:54:20,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:54:20,924 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-15 02:54:20,931 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-15 02:54:20,931 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-15 02:54:20,931 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-15 02:54:20,931 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-15 02:54:20,931 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-15 02:54:20,935 INFO [RS:0;jenkins-hbase20:34681-longCompactions-0] throttle.PressureAwareThroughputController(145): b7b06ed62cbe8eca9103c3d111564e6c#B#compaction#161 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-15 02:54:20,936 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-15 02:54:20,936 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-15 02:54:20,936 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681527240182.b7b06ed62cbe8eca9103c3d111564e6c. 2023-04-15 02:54:20,937 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681527240182.b7b06ed62cbe8eca9103c3d111564e6c. started... 2023-04-15 02:54:20,937 INFO [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing b7b06ed62cbe8eca9103c3d111564e6c 3/3 column families, dataSize=60.38 KB heapSize=180.05 KB 2023-04-15 02:54:20,937 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK b7b06ed62cbe8eca9103c3d111564e6c, store=A 2023-04-15 02:54:20,937 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-15 02:54:20,937 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK b7b06ed62cbe8eca9103c3d111564e6c, store=B 2023-04-15 02:54:20,937 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-15 02:54:20,937 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK b7b06ed62cbe8eca9103c3d111564e6c, store=C 2023-04-15 02:54:20,937 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-15 02:54:20,943 INFO [RS:0;jenkins-hbase20:34681-shortCompactions-0] throttle.PressureAwareThroughputController(145): b7b06ed62cbe8eca9103c3d111564e6c#A#compaction#162 average throughput is 2.82 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-15 02:54:20,955 DEBUG [RS:0;jenkins-hbase20:34681-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/b7b06ed62cbe8eca9103c3d111564e6c/.tmp/B/6775e098b8ff4b198521a7e961476a2b as hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/b7b06ed62cbe8eca9103c3d111564e6c/B/6775e098b8ff4b198521a7e961476a2b 2023-04-15 02:54:20,960 INFO [RS:0;jenkins-hbase20:34681-longCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in b7b06ed62cbe8eca9103c3d111564e6c/B of b7b06ed62cbe8eca9103c3d111564e6c into 6775e098b8ff4b198521a7e961476a2b(size=12.6 K), total size for store is 12.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-15 02:54:20,960 DEBUG [RS:0;jenkins-hbase20:34681-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for b7b06ed62cbe8eca9103c3d111564e6c: 2023-04-15 02:54:20,961 INFO [RS:0;jenkins-hbase20:34681-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681527240182.b7b06ed62cbe8eca9103c3d111564e6c., storeName=b7b06ed62cbe8eca9103c3d111564e6c/B, priority=11, startTime=1681527260913; duration=0sec 2023-04-15 02:54:20,961 DEBUG [RS:0;jenkins-hbase20:34681-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-15 02:54:20,961 DEBUG [RS:0;jenkins-hbase20:34681-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-04-15 02:54:20,965 DEBUG [RS:0;jenkins-hbase20:34681-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 61408 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-04-15 02:54:20,965 DEBUG [RS:0;jenkins-hbase20:34681-longCompactions-0] regionserver.HStore(1912): b7b06ed62cbe8eca9103c3d111564e6c/C is initiating minor compaction (all files) 2023-04-15 02:54:20,965 INFO [RS:0;jenkins-hbase20:34681-longCompactions-0] regionserver.HRegion(2259): Starting compaction of b7b06ed62cbe8eca9103c3d111564e6c/C in TestAcidGuarantees,,1681527240182.b7b06ed62cbe8eca9103c3d111564e6c. 2023-04-15 02:54:20,965 INFO [RS:0;jenkins-hbase20:34681-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/b7b06ed62cbe8eca9103c3d111564e6c/C/24768a1fbb7943f48b50e45cb17b549e, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/b7b06ed62cbe8eca9103c3d111564e6c/C/4b3fbff5251742bbb1852d90743ca06f, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/b7b06ed62cbe8eca9103c3d111564e6c/C/c99c8c8578d84adbb4eb965d5c423d59, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/b7b06ed62cbe8eca9103c3d111564e6c/C/71abddb901ed4f5bb7ff4e17b9d0cf66, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/b7b06ed62cbe8eca9103c3d111564e6c/C/65fa15372bed4cf0b6d1ef5b6d16d9cd] into tmpdir=hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/b7b06ed62cbe8eca9103c3d111564e6c/.tmp, totalSize=60.0 K 2023-04-15 02:54:20,966 DEBUG [RS:0;jenkins-hbase20:34681-longCompactions-0] compactions.Compactor(207): Compacting 24768a1fbb7943f48b50e45cb17b549e, keycount=150, bloomtype=ROW, size=12.4 K, encoding=NONE, compression=NONE, seqNum=669, earliestPutTs=1681527254073 2023-04-15 02:54:20,967 DEBUG [RS:0;jenkins-hbase20:34681-longCompactions-0] compactions.Compactor(207): Compacting 4b3fbff5251742bbb1852d90743ca06f, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=681, earliestPutTs=1681527254092 2023-04-15 02:54:20,967 DEBUG [RS:0;jenkins-hbase20:34681-longCompactions-0] compactions.Compactor(207): Compacting c99c8c8578d84adbb4eb965d5c423d59, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=704, earliestPutTs=1681527255247 2023-04-15 02:54:20,968 DEBUG [RS:0;jenkins-hbase20:34681-longCompactions-0] compactions.Compactor(207): Compacting 71abddb901ed4f5bb7ff4e17b9d0cf66, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=716, earliestPutTs=1681527257392 2023-04-15 02:54:20,968 DEBUG [RS:0;jenkins-hbase20:34681-longCompactions-0] compactions.Compactor(207): Compacting 65fa15372bed4cf0b6d1ef5b6d16d9cd, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=737, earliestPutTs=1681527258539 2023-04-15 02:54:20,970 INFO [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230415641aa4a51611443e8dc71c064d4e6264 to hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230415641aa4a51611443e8dc71c064d4e6264 2023-04-15 02:54:20,981 INFO [RS:0;jenkins-hbase20:34681-longCompactions-0] throttle.PressureAwareThroughputController(145): b7b06ed62cbe8eca9103c3d111564e6c#C#compaction#164 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-15 02:54:21,379 INFO [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=749, memsize=20.1 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/b7b06ed62cbe8eca9103c3d111564e6c/.tmp/A/a884776d4eca436ea80b915c5481730c 2023-04-15 02:54:21,396 DEBUG [RS:0;jenkins-hbase20:34681-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/b7b06ed62cbe8eca9103c3d111564e6c/.tmp/C/6ca060a294064b6084893685e0c90407 as hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/b7b06ed62cbe8eca9103c3d111564e6c/C/6ca060a294064b6084893685e0c90407 2023-04-15 02:54:21,404 INFO [RS:0;jenkins-hbase20:34681-longCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in b7b06ed62cbe8eca9103c3d111564e6c/C of b7b06ed62cbe8eca9103c3d111564e6c into 6ca060a294064b6084893685e0c90407(size=12.5 K), total size for store is 12.5 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-15 02:54:21,404 DEBUG [RS:0;jenkins-hbase20:34681-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for b7b06ed62cbe8eca9103c3d111564e6c: 2023-04-15 02:54:21,404 INFO [RS:0;jenkins-hbase20:34681-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681527240182.b7b06ed62cbe8eca9103c3d111564e6c., storeName=b7b06ed62cbe8eca9103c3d111564e6c/C, priority=11, startTime=1681527260913; duration=0sec 2023-04-15 02:54:21,404 DEBUG [RS:0;jenkins-hbase20:34681-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-15 02:54:21,699 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34681] regionserver.HRegion(9158): Flush requested on b7b06ed62cbe8eca9103c3d111564e6c 2023-04-15 02:54:21,699 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681527240182.b7b06ed62cbe8eca9103c3d111564e6c. as already flushing 2023-04-15 02:54:21,736 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b7b06ed62cbe8eca9103c3d111564e6c, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:54:21,736 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34681] ipc.CallRunner(144): callId: 28 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:39108 deadline: 1681527321734, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b7b06ed62cbe8eca9103c3d111564e6c, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:21,738 DEBUG [Thread-854] client.RpcRetryingCallerImpl(129): Call exception, tries=8, retries=16, started=18258 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b7b06ed62cbe8eca9103c3d111564e6c, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_0' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1681527240182.b7b06ed62cbe8eca9103c3d111564e6c., hostname=jenkins-hbase20.apache.org,34681,1681527209133, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b7b06ed62cbe8eca9103c3d111564e6c, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor31.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b7b06ed62cbe8eca9103c3d111564e6c, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-15 02:54:21,740 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b7b06ed62cbe8eca9103c3d111564e6c, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:54:21,741 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34681] ipc.CallRunner(144): callId: 44 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:39116 deadline: 1681527321735, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b7b06ed62cbe8eca9103c3d111564e6c, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:21,750 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b7b06ed62cbe8eca9103c3d111564e6c, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:54:21,750 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34681] ipc.CallRunner(144): callId: 25 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:39142 deadline: 1681527321748, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b7b06ed62cbe8eca9103c3d111564e6c, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:21,752 DEBUG [Thread-856] client.RpcRetryingCallerImpl(129): Call exception, tries=8, retries=16, started=18268 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b7b06ed62cbe8eca9103c3d111564e6c, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_1' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1681527240182.b7b06ed62cbe8eca9103c3d111564e6c., hostname=jenkins-hbase20.apache.org,34681,1681527209133, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b7b06ed62cbe8eca9103c3d111564e6c, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor31.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b7b06ed62cbe8eca9103c3d111564e6c, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-15 02:54:21,798 INFO [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=749 (bloomFilter=true), to=hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/b7b06ed62cbe8eca9103c3d111564e6c/.tmp/B/c52236b4ba9443eeba5bd3afd5bb2bc6 2023-04-15 02:54:21,798 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b7b06ed62cbe8eca9103c3d111564e6c, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:54:21,798 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34681] ipc.CallRunner(144): callId: 449 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:39132 deadline: 1681527321796, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b7b06ed62cbe8eca9103c3d111564e6c, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:21,802 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b7b06ed62cbe8eca9103c3d111564e6c, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:54:21,802 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34681] ipc.CallRunner(144): callId: 421 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:39106 deadline: 1681527321799, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b7b06ed62cbe8eca9103c3d111564e6c, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:21,844 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b7b06ed62cbe8eca9103c3d111564e6c, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:54:21,844 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34681] ipc.CallRunner(144): callId: 46 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:39116 deadline: 1681527321842, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b7b06ed62cbe8eca9103c3d111564e6c, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:22,046 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b7b06ed62cbe8eca9103c3d111564e6c, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:54:22,047 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34681] ipc.CallRunner(144): callId: 48 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:39116 deadline: 1681527322045, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b7b06ed62cbe8eca9103c3d111564e6c, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:22,173 DEBUG [RS:0;jenkins-hbase20:34681-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/b7b06ed62cbe8eca9103c3d111564e6c/.tmp/A/a0b41bd1140847b3b7ae9856b322d46f as hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/b7b06ed62cbe8eca9103c3d111564e6c/A/a0b41bd1140847b3b7ae9856b322d46f 2023-04-15 02:54:22,177 INFO [RS:0;jenkins-hbase20:34681-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in b7b06ed62cbe8eca9103c3d111564e6c/A of b7b06ed62cbe8eca9103c3d111564e6c into a0b41bd1140847b3b7ae9856b322d46f(size=26.1 K), total size for store is 26.1 K. This selection was in queue for 0sec, and took 1sec to execute. 2023-04-15 02:54:22,177 DEBUG [RS:0;jenkins-hbase20:34681-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for b7b06ed62cbe8eca9103c3d111564e6c: 2023-04-15 02:54:22,177 INFO [RS:0;jenkins-hbase20:34681-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681527240182.b7b06ed62cbe8eca9103c3d111564e6c., storeName=b7b06ed62cbe8eca9103c3d111564e6c/A, priority=11, startTime=1681527260912; duration=1sec 2023-04-15 02:54:22,177 DEBUG [RS:0;jenkins-hbase20:34681-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-15 02:54:22,213 INFO [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=749 (bloomFilter=true), to=hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/b7b06ed62cbe8eca9103c3d111564e6c/.tmp/C/796ac175dd4b4d289c2b8b30ae651f10 2023-04-15 02:54:22,217 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/b7b06ed62cbe8eca9103c3d111564e6c/.tmp/A/a884776d4eca436ea80b915c5481730c as hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/b7b06ed62cbe8eca9103c3d111564e6c/A/a884776d4eca436ea80b915c5481730c 2023-04-15 02:54:22,220 INFO [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/b7b06ed62cbe8eca9103c3d111564e6c/A/a884776d4eca436ea80b915c5481730c, entries=150, sequenceid=749, filesize=25.4 K 2023-04-15 02:54:22,220 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.StoreScanner(961): StoreScanner already closing. There is no need to updateReaders 2023-04-15 02:54:22,220 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/b7b06ed62cbe8eca9103c3d111564e6c/.tmp/B/c52236b4ba9443eeba5bd3afd5bb2bc6 as hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/b7b06ed62cbe8eca9103c3d111564e6c/B/c52236b4ba9443eeba5bd3afd5bb2bc6 2023-04-15 02:54:22,224 INFO [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/b7b06ed62cbe8eca9103c3d111564e6c/B/c52236b4ba9443eeba5bd3afd5bb2bc6, entries=150, sequenceid=749, filesize=11.9 K 2023-04-15 02:54:22,225 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/b7b06ed62cbe8eca9103c3d111564e6c/.tmp/C/796ac175dd4b4d289c2b8b30ae651f10 as hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/b7b06ed62cbe8eca9103c3d111564e6c/C/796ac175dd4b4d289c2b8b30ae651f10 2023-04-15 02:54:22,232 INFO [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/b7b06ed62cbe8eca9103c3d111564e6c/C/796ac175dd4b4d289c2b8b30ae651f10, entries=150, sequenceid=749, filesize=11.9 K 2023-04-15 02:54:22,233 INFO [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~60.38 KB/61830, heapSize ~180 KB/184320, currentSize=114.05 KB/116790 for b7b06ed62cbe8eca9103c3d111564e6c in 1295ms, sequenceid=749, compaction requested=false 2023-04-15 02:54:22,233 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for b7b06ed62cbe8eca9103c3d111564e6c: 2023-04-15 02:54:22,233 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681527240182.b7b06ed62cbe8eca9103c3d111564e6c. 2023-04-15 02:54:22,233 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-15 02:54:22,233 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-15 02:54:22,233 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:22,233 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-15 02:54:22,233 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34681,1681527209133' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-15 02:54:22,234 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:22,235 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:22,235 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-15 02:54:22,235 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:22,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:22,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:54:22,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:54:22,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:54:22,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:54:22,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:54:22,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:22,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:54:22,236 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34681,1681527209133' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-15 02:54:22,236 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@695232ce[Count = 0] remaining members to acquire global barrier 2023-04-15 02:54:22,236 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-15 02:54:22,236 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:22,237 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:22,237 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:22,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:22,237 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-15 02:54:22,237 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-15 02:54:22,237 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34681,1681527209133' in zk 2023-04-15 02:54:22,238 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:22,238 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-15 02:54:22,238 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:22,238 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-15 02:54:22,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:22,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:54:22,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:54:22,238 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-15 02:54:22,238 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-15 02:54:22,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:54:22,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:54:22,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:54:22,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:22,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:54:22,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:54:22,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:22,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34681,1681527209133': 2023-04-15 02:54:22,240 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34681,1681527209133' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-15 02:54:22,240 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-15 02:54:22,240 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-15 02:54:22,240 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-15 02:54:22,240 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-15 02:54:22,240 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-15 02:54:22,241 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:22,241 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:22,241 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-15 02:54:22,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:22,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:54:22,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:54:22,241 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:22,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:22,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:54:22,242 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:22,242 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-15 02:54:22,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-15 02:54:22,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:22,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:54:22,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:54:22,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:54:22,243 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:22,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:22,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:54:22,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:54:22,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:22,245 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:22,245 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:54:22,245 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-15 02:54:22,245 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:54:22,245 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-15 02:54:22,245 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-15 02:54:22,245 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-15 02:54:22,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-15 02:54:22,245 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-15 02:54:22,245 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-15 02:54:22,245 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:22,245 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:22,245 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:22,245 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:22,245 DEBUG [Thread-864] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-15 02:54:22,245 DEBUG [Thread-864] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-15 02:54:22,245 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-15 02:54:22,245 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-15 02:54:22,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-15 02:54:22,345 DEBUG [Thread-864] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-15 02:54:22,346 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-15 02:54:22,347 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-15 02:54:22,348 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-15 02:54:22,348 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-15 02:54:22,348 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-15 02:54:22,348 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-15 02:54:22,348 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-15 02:54:22,348 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-15 02:54:22,348 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:22,348 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:54:22,349 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-15 02:54:22,349 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-15 02:54:22,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-15 02:54:22,349 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:22,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:54:22,349 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:22,349 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-15 02:54:22,349 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:22,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-15 02:54:22,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:54:22,350 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-15 02:54:22,350 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-15 02:54:22,350 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-15 02:54:22,350 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-15 02:54:22,350 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-15 02:54:22,350 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-15 02:54:22,350 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-15 02:54:22,350 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-15 02:54:22,350 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681527240182.b7b06ed62cbe8eca9103c3d111564e6c. 2023-04-15 02:54:22,350 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681527240182.b7b06ed62cbe8eca9103c3d111564e6c. started... 2023-04-15 02:54:22,350 INFO [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing b7b06ed62cbe8eca9103c3d111564e6c 3/3 column families, dataSize=114.05 KB heapSize=339.66 KB 2023-04-15 02:54:22,351 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34681] regionserver.HRegion(9158): Flush requested on b7b06ed62cbe8eca9103c3d111564e6c 2023-04-15 02:54:22,351 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681527261717, max timestamp=1681527262350Num uniques 859; 2023-04-15 02:54:22,351 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-3] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-15 02:54:22,351 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681527240182.b7b06ed62cbe8eca9103c3d111564e6c. as already flushing 2023-04-15 02:54:22,352 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681527261717, max timestamp=1681527262350Num uniques 859; 2023-04-15 02:54:22,352 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-15 02:54:22,352 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681527261717, max timestamp=1681527262350Num uniques 859; 2023-04-15 02:54:22,352 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-5] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-15 02:54:22,352 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK b7b06ed62cbe8eca9103c3d111564e6c, store=A 2023-04-15 02:54:22,352 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-15 02:54:22,352 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK b7b06ed62cbe8eca9103c3d111564e6c, store=B 2023-04-15 02:54:22,352 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-15 02:54:22,352 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK b7b06ed62cbe8eca9103c3d111564e6c, store=C 2023-04-15 02:54:22,352 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-15 02:54:22,387 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1681527262374, max timestamp=1681527262385Num uniques 859; 2023-04-15 02:54:22,387 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1681527262374, max timestamp=1681527262385Num uniques 859; 2023-04-15 02:54:22,387 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1681527262374, max timestamp=1681527262385Num uniques 859; 2023-04-15 02:54:22,387 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-1] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-04-15 02:54:22,387 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-8] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-04-15 02:54:22,387 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-2] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-04-15 02:54:22,429 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=168, cellSize=7688, totalHeapSize=17712, min timestamp=1681527262399, max timestamp=1681527262427Num uniques 1118; 2023-04-15 02:54:22,429 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-7] regionserver.CompactionPipeline(163): Suffix data size=51198, new segment data size=7688, suffix heap size=144240,new segment heap size=17712 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1118, new segment cells count=168 2023-04-15 02:54:22,430 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=168, cellSize=7688, totalHeapSize=17712, min timestamp=1681527262399, max timestamp=1681527262427Num uniques 1118; 2023-04-15 02:54:22,430 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=168, cellSize=7688, totalHeapSize=17712, min timestamp=1681527262399, max timestamp=1681527262427Num uniques 1118; 2023-04-15 02:54:22,430 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-6] regionserver.CompactionPipeline(163): Suffix data size=51198, new segment data size=7688, suffix heap size=144240,new segment heap size=17712 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1118, new segment cells count=168 2023-04-15 02:54:22,430 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-9] regionserver.CompactionPipeline(163): Suffix data size=51198, new segment data size=7688, suffix heap size=144240,new segment heap size=17712 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1118, new segment cells count=168 2023-04-15 02:54:22,471 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=277, cellSize=12682, totalHeapSize=29048, min timestamp=1681527262439, max timestamp=1681527262469Num uniques 1027; 2023-04-15 02:54:22,471 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-3] regionserver.CompactionPipeline(163): Suffix data size=47032, new segment data size=12682, suffix heap size=134776,new segment heap size=29048 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1027, new segment cells count=277 2023-04-15 02:54:22,472 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=277, cellSize=12682, totalHeapSize=29048, min timestamp=1681527262439, max timestamp=1681527262469Num uniques 1027; 2023-04-15 02:54:22,472 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-4] regionserver.CompactionPipeline(163): Suffix data size=47032, new segment data size=12682, suffix heap size=134776,new segment heap size=29048 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1027, new segment cells count=277 2023-04-15 02:54:22,472 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=277, cellSize=12682, totalHeapSize=29048, min timestamp=1681527262439, max timestamp=1681527262469Num uniques 1027; 2023-04-15 02:54:22,472 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(163): Suffix data size=47032, new segment data size=12682, suffix heap size=134776,new segment heap size=29048 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1027, new segment cells count=277 2023-04-15 02:54:22,516 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=186, cellSize=8516, totalHeapSize=19584, min timestamp=1681527262507, max timestamp=1681527262514Num uniques 1136; 2023-04-15 02:54:22,516 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-1] regionserver.CompactionPipeline(163): Suffix data size=52026, new segment data size=8516, suffix heap size=146112,new segment heap size=19584 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1136, new segment cells count=186 2023-04-15 02:54:22,516 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=186, cellSize=8516, totalHeapSize=19584, min timestamp=1681527262507, max timestamp=1681527262514Num uniques 1136; 2023-04-15 02:54:22,516 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-5] regionserver.CompactionPipeline(163): Suffix data size=52026, new segment data size=8516, suffix heap size=146112,new segment heap size=19584 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1136, new segment cells count=186 2023-04-15 02:54:22,517 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=186, cellSize=8516, totalHeapSize=19584, min timestamp=1681527262507, max timestamp=1681527262514Num uniques 1136; 2023-04-15 02:54:22,517 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-8] regionserver.CompactionPipeline(163): Suffix data size=52026, new segment data size=8516, suffix heap size=146112,new segment heap size=19584 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1136, new segment cells count=186 2023-04-15 02:54:22,554 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=195, cellSize=8930, totalHeapSize=20520, min timestamp=1681527262543, max timestamp=1681527262552Num uniques 1045; 2023-04-15 02:54:22,554 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-7] regionserver.CompactionPipeline(163): Suffix data size=47860, new segment data size=8930, suffix heap size=136648,new segment heap size=20520 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1045, new segment cells count=195 2023-04-15 02:54:22,554 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=195, cellSize=8930, totalHeapSize=20520, min timestamp=1681527262543, max timestamp=1681527262552Num uniques 1045; 2023-04-15 02:54:22,554 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-2] regionserver.CompactionPipeline(163): Suffix data size=47860, new segment data size=8930, suffix heap size=136648,new segment heap size=20520 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1045, new segment cells count=195 2023-04-15 02:54:22,555 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=195, cellSize=8930, totalHeapSize=20520, min timestamp=1681527262543, max timestamp=1681527262552Num uniques 1045; 2023-04-15 02:54:22,555 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-6] regionserver.CompactionPipeline(163): Suffix data size=47860, new segment data size=8930, suffix heap size=136648,new segment heap size=20520 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1045, new segment cells count=195 2023-04-15 02:54:22,607 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=154, cellSize=7050, totalHeapSize=16256, min timestamp=1681527262593, max timestamp=1681527262606Num uniques 1054; 2023-04-15 02:54:22,607 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-3] regionserver.CompactionPipeline(163): Suffix data size=48270, new segment data size=7050, suffix heap size=137584,new segment heap size=16256 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1054, new segment cells count=154 2023-04-15 02:54:22,608 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=154, cellSize=7050, totalHeapSize=16256, min timestamp=1681527262593, max timestamp=1681527262606Num uniques 1054; 2023-04-15 02:54:22,608 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-4] regionserver.CompactionPipeline(163): Suffix data size=48270, new segment data size=7050, suffix heap size=137584,new segment heap size=16256 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1054, new segment cells count=154 2023-04-15 02:54:22,608 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=154, cellSize=7050, totalHeapSize=16256, min timestamp=1681527262593, max timestamp=1681527262606Num uniques 1054; 2023-04-15 02:54:22,608 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-9] regionserver.CompactionPipeline(163): Suffix data size=48270, new segment data size=7050, suffix heap size=137584,new segment heap size=16256 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1054, new segment cells count=154 2023-04-15 02:54:22,659 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=263, cellSize=12038, totalHeapSize=27592, min timestamp=1681527262642, max timestamp=1681527262657Num uniques 1013; 2023-04-15 02:54:22,659 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(163): Suffix data size=46388, new segment data size=12038, suffix heap size=133320,new segment heap size=27592 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1013, new segment cells count=263 2023-04-15 02:54:22,659 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=263, cellSize=12038, totalHeapSize=27592, min timestamp=1681527262642, max timestamp=1681527262657Num uniques 1013; 2023-04-15 02:54:22,659 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=263, cellSize=12038, totalHeapSize=27592, min timestamp=1681527262642, max timestamp=1681527262657Num uniques 1013; 2023-04-15 02:54:22,660 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-1] regionserver.CompactionPipeline(163): Suffix data size=46388, new segment data size=12038, suffix heap size=133320,new segment heap size=27592 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1013, new segment cells count=263 2023-04-15 02:54:22,660 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-5] regionserver.CompactionPipeline(163): Suffix data size=46388, new segment data size=12038, suffix heap size=133320,new segment heap size=27592 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1013, new segment cells count=263 2023-04-15 02:54:22,714 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=172, cellSize=7872, totalHeapSize=18128, min timestamp=1681527262698, max timestamp=1681527262712Num uniques 1122; 2023-04-15 02:54:22,714 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-2] regionserver.CompactionPipeline(163): Suffix data size=51382, new segment data size=7872, suffix heap size=144656,new segment heap size=18128 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1122, new segment cells count=172 2023-04-15 02:54:22,715 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=172, cellSize=7872, totalHeapSize=18128, min timestamp=1681527262698, max timestamp=1681527262712Num uniques 1122; 2023-04-15 02:54:22,715 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=172, cellSize=7872, totalHeapSize=18128, min timestamp=1681527262698, max timestamp=1681527262712Num uniques 1122; 2023-04-15 02:54:22,715 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-8] regionserver.CompactionPipeline(163): Suffix data size=51382, new segment data size=7872, suffix heap size=144656,new segment heap size=18128 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1122, new segment cells count=172 2023-04-15 02:54:22,715 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-7] regionserver.CompactionPipeline(163): Suffix data size=51382, new segment data size=7872, suffix heap size=144656,new segment heap size=18128 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1122, new segment cells count=172 2023-04-15 02:54:22,766 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=231, cellSize=10576, totalHeapSize=24264, min timestamp=1681527262750, max timestamp=1681527262763Num uniques 1031; 2023-04-15 02:54:22,766 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-6] regionserver.CompactionPipeline(163): Suffix data size=47216, new segment data size=10576, suffix heap size=135192,new segment heap size=24264 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1031, new segment cells count=231 2023-04-15 02:54:22,766 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=231, cellSize=10576, totalHeapSize=24264, min timestamp=1681527262750, max timestamp=1681527262763Num uniques 1031; 2023-04-15 02:54:22,766 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-3] regionserver.CompactionPipeline(163): Suffix data size=47216, new segment data size=10576, suffix heap size=135192,new segment heap size=24264 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1031, new segment cells count=231 2023-04-15 02:54:22,766 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=181, cellSize=8286, totalHeapSize=19064, min timestamp=1681527262754, max timestamp=1681527262763Num uniques 1031; 2023-04-15 02:54:22,766 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-4] regionserver.CompactionPipeline(163): Suffix data size=47216, new segment data size=8286, suffix heap size=135192,new segment heap size=19064 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1031, new segment cells count=181 2023-04-15 02:54:22,772 INFO [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304156f45af92fd8b4759a0507d03cb177d33 to hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304156f45af92fd8b4759a0507d03cb177d33 2023-04-15 02:54:22,778 INFO [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=773, memsize=8.9 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/b7b06ed62cbe8eca9103c3d111564e6c/.tmp/A/5f60f982270942989c1ef49b647c6b9f 2023-04-15 02:54:22,819 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=240, cellSize=10990, totalHeapSize=25200, min timestamp=1681527262805, max timestamp=1681527262817Num uniques 1090; 2023-04-15 02:54:22,819 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-9] regionserver.CompactionPipeline(163): Suffix data size=49920, new segment data size=10990, suffix heap size=141328,new segment heap size=25200 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1090, new segment cells count=240 2023-04-15 02:54:22,820 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=240, cellSize=10990, totalHeapSize=25200, min timestamp=1681527262805, max timestamp=1681527262817Num uniques 1090; 2023-04-15 02:54:22,820 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(163): Suffix data size=49920, new segment data size=10990, suffix heap size=141328,new segment heap size=25200 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1090, new segment cells count=240 2023-04-15 02:54:22,820 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=240, cellSize=10990, totalHeapSize=25200, min timestamp=1681527262805, max timestamp=1681527262817Num uniques 1040; 2023-04-15 02:54:22,834 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-1] regionserver.CompactionPipeline(163): Suffix data size=47630, new segment data size=10990, suffix heap size=136128,new segment heap size=25200 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1040, new segment cells count=240 2023-04-15 02:54:22,877 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=299, cellSize=13694, totalHeapSize=31336, min timestamp=1681527262853, max timestamp=1681527262875Num uniques 1099; 2023-04-15 02:54:22,877 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-5] regionserver.CompactionPipeline(163): Suffix data size=50334, new segment data size=13694, suffix heap size=142264,new segment heap size=31336 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1099, new segment cells count=299 2023-04-15 02:54:22,877 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=299, cellSize=13694, totalHeapSize=31336, min timestamp=1681527262853, max timestamp=1681527262875Num uniques 1099; 2023-04-15 02:54:22,877 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-8] regionserver.CompactionPipeline(163): Suffix data size=50334, new segment data size=13694, suffix heap size=142264,new segment heap size=31336 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1099, new segment cells count=299 2023-04-15 02:54:22,878 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=299, cellSize=13694, totalHeapSize=31336, min timestamp=1681527262853, max timestamp=1681527262875Num uniques 1099; 2023-04-15 02:54:22,878 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-2] regionserver.CompactionPipeline(163): Suffix data size=50334, new segment data size=13694, suffix heap size=142264,new segment heap size=31336 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1099, new segment cells count=299 2023-04-15 02:54:22,914 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b7b06ed62cbe8eca9103c3d111564e6c, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:54:22,914 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34681] ipc.CallRunner(144): callId: 258 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:39116 deadline: 1681527322914, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b7b06ed62cbe8eca9103c3d111564e6c, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:22,914 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=208, cellSize=9520, totalHeapSize=21872, min timestamp=1681527262899, max timestamp=1681527262913Num uniques 1158; 2023-04-15 02:54:22,915 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-6] regionserver.CompactionPipeline(163): Suffix data size=53030, new segment data size=9520, suffix heap size=148400,new segment heap size=21872 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1158, new segment cells count=208 2023-04-15 02:54:22,915 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=208, cellSize=9520, totalHeapSize=21872, min timestamp=1681527262899, max timestamp=1681527262913Num uniques 1158; 2023-04-15 02:54:22,915 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-7] regionserver.CompactionPipeline(163): Suffix data size=53030, new segment data size=9520, suffix heap size=148400,new segment heap size=21872 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1158, new segment cells count=208 2023-04-15 02:54:22,916 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=208, cellSize=9520, totalHeapSize=21872, min timestamp=1681527262899, max timestamp=1681527262913Num uniques 1158; 2023-04-15 02:54:22,916 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-6] regionserver.CompactionPipeline(163): Suffix data size=53030, new segment data size=9520, suffix heap size=148400,new segment heap size=21872 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1158, new segment cells count=208 2023-04-15 02:54:23,061 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=267, cellSize=12222, totalHeapSize=28008, min timestamp=1681527263026, max timestamp=1681527263060Num uniques 1067; 2023-04-15 02:54:23,061 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(163): Suffix data size=48862, new segment data size=12222, suffix heap size=138936,new segment heap size=28008 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1067, new segment cells count=267 2023-04-15 02:54:23,062 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=267, cellSize=12222, totalHeapSize=28008, min timestamp=1681527263026, max timestamp=1681527263060Num uniques 1067; 2023-04-15 02:54:23,062 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=267, cellSize=12222, totalHeapSize=28008, min timestamp=1681527263026, max timestamp=1681527263060Num uniques 1067; 2023-04-15 02:54:23,062 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-9] regionserver.CompactionPipeline(163): Suffix data size=48862, new segment data size=12222, suffix heap size=138936,new segment heap size=28008 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1067, new segment cells count=267 2023-04-15 02:54:23,062 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-4] regionserver.CompactionPipeline(163): Suffix data size=48862, new segment data size=12222, suffix heap size=138936,new segment heap size=28008 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1067, new segment cells count=267 2023-04-15 02:54:23,101 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=226, cellSize=10346, totalHeapSize=23744, min timestamp=1681527263087, max timestamp=1681527263099Num uniques 1126; 2023-04-15 02:54:23,101 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-8] regionserver.CompactionPipeline(163): Suffix data size=51566, new segment data size=10346, suffix heap size=145072,new segment heap size=23744 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1126, new segment cells count=226 2023-04-15 02:54:23,102 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=226, cellSize=10346, totalHeapSize=23744, min timestamp=1681527263087, max timestamp=1681527263099Num uniques 1126; 2023-04-15 02:54:23,102 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=226, cellSize=10346, totalHeapSize=23744, min timestamp=1681527263087, max timestamp=1681527263099Num uniques 1126; 2023-04-15 02:54:23,102 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-5] regionserver.CompactionPipeline(163): Suffix data size=51566, new segment data size=10346, suffix heap size=145072,new segment heap size=23744 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1126, new segment cells count=226 2023-04-15 02:54:23,102 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-1] regionserver.CompactionPipeline(163): Suffix data size=51566, new segment data size=10346, suffix heap size=145072,new segment heap size=23744 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1126, new segment cells count=226 2023-04-15 02:54:23,139 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=185, cellSize=8470, totalHeapSize=19480, min timestamp=1681527263118, max timestamp=1681527263137Num uniques 1085; 2023-04-15 02:54:23,139 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-7] regionserver.CompactionPipeline(163): Suffix data size=49690, new segment data size=8470, suffix heap size=140808,new segment heap size=19480 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1085, new segment cells count=185 2023-04-15 02:54:23,139 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=185, cellSize=8470, totalHeapSize=19480, min timestamp=1681527263118, max timestamp=1681527263137Num uniques 1085; 2023-04-15 02:54:23,139 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-2] regionserver.CompactionPipeline(163): Suffix data size=49690, new segment data size=8470, suffix heap size=140808,new segment heap size=19480 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1085, new segment cells count=185 2023-04-15 02:54:23,139 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=185, cellSize=8470, totalHeapSize=19480, min timestamp=1681527263118, max timestamp=1681527263137Num uniques 1085; 2023-04-15 02:54:23,140 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-6] regionserver.CompactionPipeline(163): Suffix data size=49690, new segment data size=8470, suffix heap size=140808,new segment heap size=19480 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1085, new segment cells count=185 2023-04-15 02:54:23,183 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=244, cellSize=11174, totalHeapSize=25616, min timestamp=1681527263166, max timestamp=1681527263180Num uniques 1044; 2023-04-15 02:54:23,183 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=244, cellSize=11174, totalHeapSize=25616, min timestamp=1681527263166, max timestamp=1681527263180Num uniques 1044; 2023-04-15 02:54:23,183 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=194, cellSize=8884, totalHeapSize=20416, min timestamp=1681527263169, max timestamp=1681527263180Num uniques 1044; 2023-04-15 02:54:23,183 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(163): Suffix data size=47814, new segment data size=11174, suffix heap size=136544,new segment heap size=25616 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1044, new segment cells count=244 2023-04-15 02:54:23,183 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-3] regionserver.CompactionPipeline(163): Suffix data size=47814, new segment data size=11174, suffix heap size=136544,new segment heap size=25616 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1044, new segment cells count=244 2023-04-15 02:54:23,183 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-9] regionserver.CompactionPipeline(163): Suffix data size=47814, new segment data size=8884, suffix heap size=136544,new segment heap size=20416 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1044, new segment cells count=194 2023-04-15 02:54:23,196 INFO [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=773 (bloomFilter=true), to=hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/b7b06ed62cbe8eca9103c3d111564e6c/.tmp/B/d865f93b64654bee8122523c963aa77b 2023-04-15 02:54:23,231 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=253, cellSize=11585, totalHeapSize=26552, min timestamp=1681527263215, max timestamp=1681527263229Num uniques 1103; 2023-04-15 02:54:23,232 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=253, cellSize=11585, totalHeapSize=26552, min timestamp=1681527263215, max timestamp=1681527263229Num uniques 1053; 2023-04-15 02:54:23,231 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=253, cellSize=11585, totalHeapSize=26552, min timestamp=1681527263215, max timestamp=1681527263229Num uniques 1103; 2023-04-15 02:54:23,232 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-5] regionserver.CompactionPipeline(163): Suffix data size=48225, new segment data size=11585, suffix heap size=137480,new segment heap size=26552 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1053, new segment cells count=253 2023-04-15 02:54:23,232 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-8] regionserver.CompactionPipeline(163): Suffix data size=50515, new segment data size=11585, suffix heap size=142680,new segment heap size=26552 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1103, new segment cells count=253 2023-04-15 02:54:23,232 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-4] regionserver.CompactionPipeline(163): Suffix data size=50515, new segment data size=11585, suffix heap size=142680,new segment heap size=26552 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1103, new segment cells count=253 2023-04-15 02:54:23,269 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=212, cellSize=9702, totalHeapSize=22288, min timestamp=1681527263256, max timestamp=1681527263268Num uniques 1112; 2023-04-15 02:54:23,269 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=212, cellSize=9702, totalHeapSize=22288, min timestamp=1681527263256, max timestamp=1681527263268Num uniques 1112; 2023-04-15 02:54:23,270 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-1] regionserver.CompactionPipeline(163): Suffix data size=50922, new segment data size=9702, suffix heap size=143616,new segment heap size=22288 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1112, new segment cells count=212 2023-04-15 02:54:23,270 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-7] regionserver.CompactionPipeline(163): Suffix data size=50922, new segment data size=9702, suffix heap size=143616,new segment heap size=22288 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1112, new segment cells count=212 2023-04-15 02:54:23,271 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=212, cellSize=9702, totalHeapSize=22288, min timestamp=1681527263256, max timestamp=1681527263268Num uniques 1112; 2023-04-15 02:54:23,271 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-2] regionserver.CompactionPipeline(163): Suffix data size=50922, new segment data size=9702, suffix heap size=143616,new segment heap size=22288 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1112, new segment cells count=212 2023-04-15 02:54:23,313 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=271, cellSize=12406, totalHeapSize=28424, min timestamp=1681527263296, max timestamp=1681527263311Num uniques 1071; 2023-04-15 02:54:23,313 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-6] regionserver.CompactionPipeline(163): Suffix data size=49046, new segment data size=12406, suffix heap size=139352,new segment heap size=28424 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1071, new segment cells count=271 2023-04-15 02:54:23,314 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=271, cellSize=12406, totalHeapSize=28424, min timestamp=1681527263296, max timestamp=1681527263311Num uniques 1071; 2023-04-15 02:54:23,314 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=271, cellSize=12406, totalHeapSize=28424, min timestamp=1681527263296, max timestamp=1681527263311Num uniques 1071; 2023-04-15 02:54:23,314 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(163): Suffix data size=49046, new segment data size=12406, suffix heap size=139352,new segment heap size=28424 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1071, new segment cells count=271 2023-04-15 02:54:23,314 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-3] regionserver.CompactionPipeline(163): Suffix data size=49046, new segment data size=12406, suffix heap size=139352,new segment heap size=28424 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1071, new segment cells count=271 2023-04-15 02:54:23,358 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=180, cellSize=8240, totalHeapSize=18960, min timestamp=1681527263343, max timestamp=1681527263356Num uniques 1130; 2023-04-15 02:54:23,358 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-8] regionserver.CompactionPipeline(163): Suffix data size=51750, new segment data size=8240, suffix heap size=145488,new segment heap size=18960 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1130, new segment cells count=180 2023-04-15 02:54:23,359 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=180, cellSize=8240, totalHeapSize=18960, min timestamp=1681527263343, max timestamp=1681527263356Num uniques 1130; 2023-04-15 02:54:23,359 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-9] regionserver.CompactionPipeline(163): Suffix data size=51750, new segment data size=8240, suffix heap size=145488,new segment heap size=18960 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1130, new segment cells count=180 2023-04-15 02:54:23,359 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=180, cellSize=8240, totalHeapSize=18960, min timestamp=1681527263343, max timestamp=1681527263356Num uniques 1130; 2023-04-15 02:54:23,359 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-5] regionserver.CompactionPipeline(163): Suffix data size=51750, new segment data size=8240, suffix heap size=145488,new segment heap size=18960 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1130, new segment cells count=180 2023-04-15 02:54:23,388 DEBUG [Thread-869] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x50feca84 to 127.0.0.1:51762 2023-04-15 02:54:23,388 DEBUG [Thread-869] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-15 02:54:23,389 DEBUG [Thread-865] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x61ad5d1c to 127.0.0.1:51762 2023-04-15 02:54:23,389 DEBUG [Thread-865] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-15 02:54:23,391 DEBUG [Thread-867] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x486fba63 to 127.0.0.1:51762 2023-04-15 02:54:23,391 DEBUG [Thread-867] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-15 02:54:23,391 DEBUG [Thread-873] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x0bd79dff to 127.0.0.1:51762 2023-04-15 02:54:23,391 DEBUG [Thread-873] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-15 02:54:23,396 DEBUG [Thread-858] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x2d781def to 127.0.0.1:51762 2023-04-15 02:54:23,396 DEBUG [Thread-858] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-15 02:54:23,607 INFO [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=773 (bloomFilter=true), to=hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/b7b06ed62cbe8eca9103c3d111564e6c/.tmp/C/24432b816dbe47a08e12696638836f51 2023-04-15 02:54:23,616 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/b7b06ed62cbe8eca9103c3d111564e6c/.tmp/A/5f60f982270942989c1ef49b647c6b9f as hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/b7b06ed62cbe8eca9103c3d111564e6c/A/5f60f982270942989c1ef49b647c6b9f 2023-04-15 02:54:23,624 INFO [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/b7b06ed62cbe8eca9103c3d111564e6c/A/5f60f982270942989c1ef49b647c6b9f, entries=200, sequenceid=773, filesize=32.3 K 2023-04-15 02:54:23,625 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/b7b06ed62cbe8eca9103c3d111564e6c/.tmp/B/d865f93b64654bee8122523c963aa77b as hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/b7b06ed62cbe8eca9103c3d111564e6c/B/d865f93b64654bee8122523c963aa77b 2023-04-15 02:54:23,632 INFO [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/b7b06ed62cbe8eca9103c3d111564e6c/B/d865f93b64654bee8122523c963aa77b, entries=150, sequenceid=773, filesize=11.9 K 2023-04-15 02:54:23,633 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/b7b06ed62cbe8eca9103c3d111564e6c/.tmp/C/24432b816dbe47a08e12696638836f51 as hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/b7b06ed62cbe8eca9103c3d111564e6c/C/24432b816dbe47a08e12696638836f51 2023-04-15 02:54:23,639 INFO [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/b7b06ed62cbe8eca9103c3d111564e6c/C/24432b816dbe47a08e12696638836f51, entries=150, sequenceid=773, filesize=11.9 K 2023-04-15 02:54:23,640 INFO [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~26.84 KB/27480, heapSize ~66.19 KB/67776, currentSize=93.93 KB/96180 for b7b06ed62cbe8eca9103c3d111564e6c in 1290ms, sequenceid=773, compaction requested=true 2023-04-15 02:54:23,640 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for b7b06ed62cbe8eca9103c3d111564e6c: 2023-04-15 02:54:23,640 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681527240182.b7b06ed62cbe8eca9103c3d111564e6c. 2023-04-15 02:54:23,640 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-15 02:54:23,640 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-15 02:54:23,640 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:23,640 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-15 02:54:23,640 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34681,1681527209133' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-15 02:54:23,642 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:23,642 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:23,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:23,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:54:23,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:54:23,642 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:23,642 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-15 02:54:23,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:54:23,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:54:23,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:54:23,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:23,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:54:23,644 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34681,1681527209133' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-15 02:54:23,644 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7e81001e[Count = 0] remaining members to acquire global barrier 2023-04-15 02:54:23,644 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-15 02:54:23,644 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:23,645 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:23,645 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:23,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:23,645 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-15 02:54:23,645 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-15 02:54:23,645 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34681,1681527209133' in zk 2023-04-15 02:54:23,645 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:23,645 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-15 02:54:23,646 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-15 02:54:23,646 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-15 02:54:23,646 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:23,646 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-15 02:54:23,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:23,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:54:23,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:54:23,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:54:23,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:54:23,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:54:23,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:23,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:54:23,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:54:23,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:23,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34681,1681527209133': 2023-04-15 02:54:23,649 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34681,1681527209133' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-15 02:54:23,649 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-15 02:54:23,649 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-15 02:54:23,649 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-15 02:54:23,649 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-15 02:54:23,649 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-15 02:54:23,651 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:23,651 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-15 02:54:23,651 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:23,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:23,651 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:23,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:23,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:54:23,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:54:23,651 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-15 02:54:23,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-15 02:54:23,652 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:23,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:54:23,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:23,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:54:23,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:54:23,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:54:23,653 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:23,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:23,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:54:23,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:54:23,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:23,656 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:23,656 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-15 02:54:23,656 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:54:23,656 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-15 02:54:23,656 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-15 02:54:23,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-15 02:54:23,656 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-15 02:54:23,656 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-15 02:54:23,656 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-15 02:54:23,656 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-15 02:54:23,656 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:54:23,656 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-15 02:54:23,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-15 02:54:23,656 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:23,656 DEBUG [Thread-864] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-15 02:54:23,657 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:23,657 DEBUG [Thread-864] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-15 02:54:23,657 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:23,657 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:23,757 DEBUG [Thread-864] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-15 02:54:23,758 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-15 02:54:23,806 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34681] regionserver.HRegion(9158): Flush requested on b7b06ed62cbe8eca9103c3d111564e6c 2023-04-15 02:54:23,807 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing b7b06ed62cbe8eca9103c3d111564e6c 3/3 column families, dataSize=100.63 KB heapSize=283.41 KB 2023-04-15 02:54:23,807 DEBUG [Thread-860] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x5649f410 to 127.0.0.1:51762 2023-04-15 02:54:23,807 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK b7b06ed62cbe8eca9103c3d111564e6c, store=A 2023-04-15 02:54:23,807 DEBUG [Thread-860] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-15 02:54:23,807 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-15 02:54:23,808 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK b7b06ed62cbe8eca9103c3d111564e6c, store=B 2023-04-15 02:54:23,808 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-15 02:54:23,808 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK b7b06ed62cbe8eca9103c3d111564e6c, store=C 2023-04-15 02:54:23,808 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-15 02:54:23,825 DEBUG [Thread-862] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x7acc1387 to 127.0.0.1:51762 2023-04-15 02:54:23,825 DEBUG [Thread-862] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-15 02:54:24,235 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230415974936be982245069df25bf634423c28 to hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230415974936be982245069df25bf634423c28 2023-04-15 02:54:24,644 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=1131, memsize=33.5 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/b7b06ed62cbe8eca9103c3d111564e6c/.tmp/A/1ddb9af0a434478a984b5c42a480ca02 2023-04-15 02:54:25,066 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=1131 (bloomFilter=true), to=hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/b7b06ed62cbe8eca9103c3d111564e6c/.tmp/B/42954d1787874cf9bb929e1471e57d4d 2023-04-15 02:54:25,485 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=1131 (bloomFilter=true), to=hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/b7b06ed62cbe8eca9103c3d111564e6c/.tmp/C/99f8c35b970844cda54d78cd5be15a43 2023-04-15 02:54:25,490 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/b7b06ed62cbe8eca9103c3d111564e6c/.tmp/A/1ddb9af0a434478a984b5c42a480ca02 as hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/b7b06ed62cbe8eca9103c3d111564e6c/A/1ddb9af0a434478a984b5c42a480ca02 2023-04-15 02:54:25,493 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/b7b06ed62cbe8eca9103c3d111564e6c/A/1ddb9af0a434478a984b5c42a480ca02, entries=150, sequenceid=1131, filesize=25.4 K 2023-04-15 02:54:25,494 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/b7b06ed62cbe8eca9103c3d111564e6c/.tmp/B/42954d1787874cf9bb929e1471e57d4d as hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/b7b06ed62cbe8eca9103c3d111564e6c/B/42954d1787874cf9bb929e1471e57d4d 2023-04-15 02:54:25,498 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/b7b06ed62cbe8eca9103c3d111564e6c/B/42954d1787874cf9bb929e1471e57d4d, entries=150, sequenceid=1131, filesize=11.9 K 2023-04-15 02:54:25,499 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/b7b06ed62cbe8eca9103c3d111564e6c/.tmp/C/99f8c35b970844cda54d78cd5be15a43 as hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/b7b06ed62cbe8eca9103c3d111564e6c/C/99f8c35b970844cda54d78cd5be15a43 2023-04-15 02:54:25,503 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/b7b06ed62cbe8eca9103c3d111564e6c/C/99f8c35b970844cda54d78cd5be15a43, entries=150, sequenceid=1131, filesize=11.9 K 2023-04-15 02:54:25,504 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~100.63 KB/103050, heapSize ~283.36 KB/290160, currentSize=6.71 KB/6870 for b7b06ed62cbe8eca9103c3d111564e6c in 1698ms, sequenceid=1131, compaction requested=true 2023-04-15 02:54:25,504 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for b7b06ed62cbe8eca9103c3d111564e6c: 2023-04-15 02:54:25,504 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-15 02:54:25,504 DEBUG [RS:0;jenkins-hbase20:34681-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-15 02:54:25,504 DEBUG [RS:0;jenkins-hbase20:34681-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-15 02:54:25,504 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-15 02:54:25,504 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-15 02:54:25,505 DEBUG [RS:0;jenkins-hbase20:34681-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 49426 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-15 02:54:25,506 DEBUG [RS:0;jenkins-hbase20:34681-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 111906 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-15 02:54:25,506 DEBUG [RS:0;jenkins-hbase20:34681-longCompactions-0] regionserver.HStore(1912): b7b06ed62cbe8eca9103c3d111564e6c/B is initiating minor compaction (all files) 2023-04-15 02:54:25,506 DEBUG [RS:0;jenkins-hbase20:34681-shortCompactions-0] regionserver.HStore(1912): b7b06ed62cbe8eca9103c3d111564e6c/A is initiating minor compaction (all files) 2023-04-15 02:54:25,506 INFO [RS:0;jenkins-hbase20:34681-longCompactions-0] regionserver.HRegion(2259): Starting compaction of b7b06ed62cbe8eca9103c3d111564e6c/B in TestAcidGuarantees,,1681527240182.b7b06ed62cbe8eca9103c3d111564e6c. 2023-04-15 02:54:25,506 INFO [RS:0;jenkins-hbase20:34681-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of b7b06ed62cbe8eca9103c3d111564e6c/A in TestAcidGuarantees,,1681527240182.b7b06ed62cbe8eca9103c3d111564e6c. 2023-04-15 02:54:25,506 INFO [RS:0;jenkins-hbase20:34681-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/b7b06ed62cbe8eca9103c3d111564e6c/B/6775e098b8ff4b198521a7e961476a2b, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/b7b06ed62cbe8eca9103c3d111564e6c/B/c52236b4ba9443eeba5bd3afd5bb2bc6, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/b7b06ed62cbe8eca9103c3d111564e6c/B/d865f93b64654bee8122523c963aa77b, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/b7b06ed62cbe8eca9103c3d111564e6c/B/42954d1787874cf9bb929e1471e57d4d] into tmpdir=hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/b7b06ed62cbe8eca9103c3d111564e6c/.tmp, totalSize=48.3 K 2023-04-15 02:54:25,506 INFO [RS:0;jenkins-hbase20:34681-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/b7b06ed62cbe8eca9103c3d111564e6c/A/a0b41bd1140847b3b7ae9856b322d46f, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/b7b06ed62cbe8eca9103c3d111564e6c/A/a884776d4eca436ea80b915c5481730c, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/b7b06ed62cbe8eca9103c3d111564e6c/A/5f60f982270942989c1ef49b647c6b9f, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/b7b06ed62cbe8eca9103c3d111564e6c/A/1ddb9af0a434478a984b5c42a480ca02] into tmpdir=hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/b7b06ed62cbe8eca9103c3d111564e6c/.tmp, totalSize=109.3 K 2023-04-15 02:54:25,506 DEBUG [RS:0;jenkins-hbase20:34681-longCompactions-0] compactions.Compactor(207): Compacting 6775e098b8ff4b198521a7e961476a2b, keycount=150, bloomtype=ROW, size=12.6 K, encoding=NONE, compression=NONE, seqNum=737, earliestPutTs=1681527258539 2023-04-15 02:54:25,506 DEBUG [RS:0;jenkins-hbase20:34681-shortCompactions-0] compactions.Compactor(207): Compacting a0b41bd1140847b3b7ae9856b322d46f, keycount=150, bloomtype=ROW, size=26.1 K, encoding=NONE, compression=NONE, seqNum=737, earliestPutTs=1681527258539 2023-04-15 02:54:25,507 DEBUG [RS:0;jenkins-hbase20:34681-longCompactions-0] compactions.Compactor(207): Compacting c52236b4ba9443eeba5bd3afd5bb2bc6, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=749, earliestPutTs=1681527259668 2023-04-15 02:54:25,507 DEBUG [RS:0;jenkins-hbase20:34681-shortCompactions-0] compactions.Compactor(207): Compacting a884776d4eca436ea80b915c5481730c, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=749, earliestPutTs=1681527259668 2023-04-15 02:54:25,507 DEBUG [RS:0;jenkins-hbase20:34681-longCompactions-0] compactions.Compactor(207): Compacting d865f93b64654bee8122523c963aa77b, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=773, earliestPutTs=1681527261724 2023-04-15 02:54:25,507 DEBUG [RS:0;jenkins-hbase20:34681-shortCompactions-0] compactions.Compactor(207): Compacting 5f60f982270942989c1ef49b647c6b9f, keycount=200, bloomtype=ROW, size=32.3 K, encoding=NONE, compression=NONE, seqNum=773, earliestPutTs=1681527261717 2023-04-15 02:54:25,507 DEBUG [RS:0;jenkins-hbase20:34681-longCompactions-0] compactions.Compactor(207): Compacting 42954d1787874cf9bb929e1471e57d4d, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=1131, earliestPutTs=1681527263384 2023-04-15 02:54:25,507 DEBUG [RS:0;jenkins-hbase20:34681-shortCompactions-0] compactions.Compactor(207): Compacting 1ddb9af0a434478a984b5c42a480ca02, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=1131, earliestPutTs=1681527263384 2023-04-15 02:54:25,523 INFO [RS:0;jenkins-hbase20:34681-longCompactions-0] throttle.PressureAwareThroughputController(145): b7b06ed62cbe8eca9103c3d111564e6c#B#compaction#174 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-15 02:54:25,527 INFO [RS:0;jenkins-hbase20:34681-shortCompactions-0] throttle.PressureAwareThroughputController(145): b7b06ed62cbe8eca9103c3d111564e6c#A#compaction#173 average throughput is 1.79 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-15 02:54:25,944 DEBUG [RS:0;jenkins-hbase20:34681-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/b7b06ed62cbe8eca9103c3d111564e6c/.tmp/B/8a2a6e4ea69545cc869d1bc0e05760b1 as hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/b7b06ed62cbe8eca9103c3d111564e6c/B/8a2a6e4ea69545cc869d1bc0e05760b1 2023-04-15 02:54:25,948 INFO [RS:0;jenkins-hbase20:34681-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in b7b06ed62cbe8eca9103c3d111564e6c/B of b7b06ed62cbe8eca9103c3d111564e6c into 8a2a6e4ea69545cc869d1bc0e05760b1(size=12.7 K), total size for store is 12.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-15 02:54:25,949 DEBUG [RS:0;jenkins-hbase20:34681-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for b7b06ed62cbe8eca9103c3d111564e6c: 2023-04-15 02:54:25,949 INFO [RS:0;jenkins-hbase20:34681-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681527240182.b7b06ed62cbe8eca9103c3d111564e6c., storeName=b7b06ed62cbe8eca9103c3d111564e6c/B, priority=12, startTime=1681527265504; duration=0sec 2023-04-15 02:54:25,949 DEBUG [RS:0;jenkins-hbase20:34681-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-15 02:54:25,949 DEBUG [RS:0;jenkins-hbase20:34681-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-15 02:54:25,950 DEBUG [RS:0;jenkins-hbase20:34681-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 49392 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-15 02:54:25,950 DEBUG [RS:0;jenkins-hbase20:34681-longCompactions-0] regionserver.HStore(1912): b7b06ed62cbe8eca9103c3d111564e6c/C is initiating minor compaction (all files) 2023-04-15 02:54:25,950 INFO [RS:0;jenkins-hbase20:34681-longCompactions-0] regionserver.HRegion(2259): Starting compaction of b7b06ed62cbe8eca9103c3d111564e6c/C in TestAcidGuarantees,,1681527240182.b7b06ed62cbe8eca9103c3d111564e6c. 2023-04-15 02:54:25,950 INFO [RS:0;jenkins-hbase20:34681-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/b7b06ed62cbe8eca9103c3d111564e6c/C/6ca060a294064b6084893685e0c90407, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/b7b06ed62cbe8eca9103c3d111564e6c/C/796ac175dd4b4d289c2b8b30ae651f10, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/b7b06ed62cbe8eca9103c3d111564e6c/C/24432b816dbe47a08e12696638836f51, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/b7b06ed62cbe8eca9103c3d111564e6c/C/99f8c35b970844cda54d78cd5be15a43] into tmpdir=hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/b7b06ed62cbe8eca9103c3d111564e6c/.tmp, totalSize=48.2 K 2023-04-15 02:54:25,951 DEBUG [RS:0;jenkins-hbase20:34681-longCompactions-0] compactions.Compactor(207): Compacting 6ca060a294064b6084893685e0c90407, keycount=150, bloomtype=ROW, size=12.5 K, encoding=NONE, compression=NONE, seqNum=737, earliestPutTs=1681527258539 2023-04-15 02:54:25,951 DEBUG [RS:0;jenkins-hbase20:34681-longCompactions-0] compactions.Compactor(207): Compacting 796ac175dd4b4d289c2b8b30ae651f10, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=749, earliestPutTs=1681527259668 2023-04-15 02:54:25,951 DEBUG [RS:0;jenkins-hbase20:34681-longCompactions-0] compactions.Compactor(207): Compacting 24432b816dbe47a08e12696638836f51, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=773, earliestPutTs=1681527261724 2023-04-15 02:54:25,952 DEBUG [RS:0;jenkins-hbase20:34681-longCompactions-0] compactions.Compactor(207): Compacting 99f8c35b970844cda54d78cd5be15a43, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=1131, earliestPutTs=1681527263384 2023-04-15 02:54:25,952 DEBUG [RS:0;jenkins-hbase20:34681-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/b7b06ed62cbe8eca9103c3d111564e6c/.tmp/A/f749db64ccee454aad6728bb945bb767 as hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/b7b06ed62cbe8eca9103c3d111564e6c/A/f749db64ccee454aad6728bb945bb767 2023-04-15 02:54:25,956 INFO [RS:0;jenkins-hbase20:34681-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in b7b06ed62cbe8eca9103c3d111564e6c/A of b7b06ed62cbe8eca9103c3d111564e6c into f749db64ccee454aad6728bb945bb767(size=26.2 K), total size for store is 26.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-15 02:54:25,956 DEBUG [RS:0;jenkins-hbase20:34681-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for b7b06ed62cbe8eca9103c3d111564e6c: 2023-04-15 02:54:25,956 INFO [RS:0;jenkins-hbase20:34681-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681527240182.b7b06ed62cbe8eca9103c3d111564e6c., storeName=b7b06ed62cbe8eca9103c3d111564e6c/A, priority=12, startTime=1681527265504; duration=0sec 2023-04-15 02:54:25,956 DEBUG [RS:0;jenkins-hbase20:34681-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-15 02:54:25,959 INFO [RS:0;jenkins-hbase20:34681-longCompactions-0] throttle.PressureAwareThroughputController(145): b7b06ed62cbe8eca9103c3d111564e6c#C#compaction#175 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-15 02:54:26,368 DEBUG [RS:0;jenkins-hbase20:34681-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/b7b06ed62cbe8eca9103c3d111564e6c/.tmp/C/8be11b0f6a924755abe77e5d979844e6 as hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/b7b06ed62cbe8eca9103c3d111564e6c/C/8be11b0f6a924755abe77e5d979844e6 2023-04-15 02:54:26,372 INFO [RS:0;jenkins-hbase20:34681-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in b7b06ed62cbe8eca9103c3d111564e6c/C of b7b06ed62cbe8eca9103c3d111564e6c into 8be11b0f6a924755abe77e5d979844e6(size=12.7 K), total size for store is 12.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-15 02:54:26,372 DEBUG [RS:0;jenkins-hbase20:34681-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for b7b06ed62cbe8eca9103c3d111564e6c: 2023-04-15 02:54:26,373 INFO [RS:0;jenkins-hbase20:34681-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681527240182.b7b06ed62cbe8eca9103c3d111564e6c., storeName=b7b06ed62cbe8eca9103c3d111564e6c/C, priority=12, startTime=1681527265504; duration=0sec 2023-04-15 02:54:26,373 DEBUG [RS:0;jenkins-hbase20:34681-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-15 02:54:30,176 DEBUG [regionserver/jenkins-hbase20:0.Chore.1] throttle.PressureAwareCompactionThroughputController(103): CompactionPressure is 0.0, tune throughput to 50.00 MB/second 2023-04-15 02:54:31,839 DEBUG [Thread-854] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x071f9a8a to 127.0.0.1:51762 2023-04-15 02:54:31,840 DEBUG [Thread-854] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-15 02:54:31,844 DEBUG [Thread-856] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x430c7474 to 127.0.0.1:51762 2023-04-15 02:54:31,844 DEBUG [Thread-856] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-15 02:54:31,844 INFO [Listener at localhost.localdomain/33693] hbase.AcidGuaranteesTestTool(392): Finished test. Writers: 2023-04-15 02:54:31,845 INFO [Listener at localhost.localdomain/33693] hbase.AcidGuaranteesTestTool(394): wrote 11 2023-04-15 02:54:31,845 INFO [Listener at localhost.localdomain/33693] hbase.AcidGuaranteesTestTool(394): wrote 8 2023-04-15 02:54:31,845 INFO [Listener at localhost.localdomain/33693] hbase.AcidGuaranteesTestTool(394): wrote 381 2023-04-15 02:54:31,845 INFO [Listener at localhost.localdomain/33693] hbase.AcidGuaranteesTestTool(394): wrote 346 2023-04-15 02:54:31,845 INFO [Listener at localhost.localdomain/33693] hbase.AcidGuaranteesTestTool(394): wrote 312 2023-04-15 02:54:31,845 INFO [Listener at localhost.localdomain/33693] hbase.AcidGuaranteesTestTool(396): Readers: 2023-04-15 02:54:31,845 INFO [Listener at localhost.localdomain/33693] hbase.AcidGuaranteesTestTool(398): read 6070 2023-04-15 02:54:31,845 INFO [Listener at localhost.localdomain/33693] hbase.AcidGuaranteesTestTool(398): read 5798 2023-04-15 02:54:31,845 INFO [Listener at localhost.localdomain/33693] hbase.AcidGuaranteesTestTool(400): Scanners: 2023-04-15 02:54:31,845 INFO [Listener at localhost.localdomain/33693] hbase.AcidGuaranteesTestTool(402): scanned 2465 2023-04-15 02:54:31,845 INFO [Listener at localhost.localdomain/33693] hbase.AcidGuaranteesTestTool(403): verified 7394 rows 2023-04-15 02:54:31,845 INFO [Listener at localhost.localdomain/33693] hbase.AcidGuaranteesTestTool(402): scanned 2481 2023-04-15 02:54:31,845 INFO [Listener at localhost.localdomain/33693] hbase.AcidGuaranteesTestTool(403): verified 7442 rows 2023-04-15 02:54:31,845 INFO [Listener at localhost.localdomain/33693] client.ConnectionImplementation(1974): Closing master protocol: MasterService 2023-04-15 02:54:31,845 DEBUG [Listener at localhost.localdomain/33693] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x35bef85d to 127.0.0.1:51762 2023-04-15 02:54:31,845 DEBUG [Listener at localhost.localdomain/33693] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-15 02:54:31,846 INFO [Listener at localhost.localdomain/33693] client.HBaseAdmin$15(890): Started disable of TestAcidGuarantees 2023-04-15 02:54:31,847 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.HMaster$11(2418): Client=jenkins//148.251.75.209 disable TestAcidGuarantees 2023-04-15 02:54:31,848 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] procedure2.ProcedureExecutor(1029): Stored pid=24, state=RUNNABLE:DISABLE_TABLE_PREPARE; DisableTableProcedure table=TestAcidGuarantees 2023-04-15 02:54:31,850 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1681527271850"}]},"ts":"1681527271850"} 2023-04-15 02:54:31,850 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(1227): Checking to see if procedure is done pid=24 2023-04-15 02:54:31,851 INFO [PEWorker-4] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=DISABLING in hbase:meta 2023-04-15 02:54:31,851 INFO [PEWorker-4] procedure.DisableTableProcedure(293): Set TestAcidGuarantees to state=DISABLING 2023-04-15 02:54:31,852 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=25, ppid=24, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=b7b06ed62cbe8eca9103c3d111564e6c, UNASSIGN}] 2023-04-15 02:54:31,853 INFO [PEWorker-3] procedure.MasterProcedureScheduler(727): Took xlock for pid=25, ppid=24, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=b7b06ed62cbe8eca9103c3d111564e6c, UNASSIGN 2023-04-15 02:54:31,853 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=25 updating hbase:meta row=b7b06ed62cbe8eca9103c3d111564e6c, regionState=CLOSING, regionLocation=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:31,854 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1681527240182.b7b06ed62cbe8eca9103c3d111564e6c.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1681527271853"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1681527271853"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1681527271853"}]},"ts":"1681527271853"} 2023-04-15 02:54:31,854 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=26, ppid=25, state=RUNNABLE; CloseRegionProcedure b7b06ed62cbe8eca9103c3d111564e6c, server=jenkins-hbase20.apache.org,34681,1681527209133}] 2023-04-15 02:54:31,952 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(1227): Checking to see if procedure is done pid=24 2023-04-15 02:54:32,007 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] handler.UnassignRegionHandler(109): Close b7b06ed62cbe8eca9103c3d111564e6c 2023-04-15 02:54:32,007 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1604): Closing b7b06ed62cbe8eca9103c3d111564e6c, disabling compactions & flushes 2023-04-15 02:54:32,007 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1681527240182.b7b06ed62cbe8eca9103c3d111564e6c. 2023-04-15 02:54:32,007 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1681527240182.b7b06ed62cbe8eca9103c3d111564e6c. 2023-04-15 02:54:32,007 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1681527240182.b7b06ed62cbe8eca9103c3d111564e6c. after waiting 0 ms 2023-04-15 02:54:32,007 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1681527240182.b7b06ed62cbe8eca9103c3d111564e6c. 2023-04-15 02:54:32,008 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(2745): Flushing b7b06ed62cbe8eca9103c3d111564e6c 3/3 column families, dataSize=20.13 KB heapSize=60.52 KB 2023-04-15 02:54:32,008 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK b7b06ed62cbe8eca9103c3d111564e6c, store=A 2023-04-15 02:54:32,008 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-15 02:54:32,008 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK b7b06ed62cbe8eca9103c3d111564e6c, store=B 2023-04-15 02:54:32,008 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-15 02:54:32,009 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK b7b06ed62cbe8eca9103c3d111564e6c, store=C 2023-04-15 02:54:32,009 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-15 02:54:32,153 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(1227): Checking to see if procedure is done pid=24 2023-04-15 02:54:32,440 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023041543f8b8a274024fc9b450f2469cd07fea to hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023041543f8b8a274024fc9b450f2469cd07fea 2023-04-15 02:54:32,454 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(1227): Checking to see if procedure is done pid=24 2023-04-15 02:54:32,848 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=1140, memsize=6.7 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/b7b06ed62cbe8eca9103c3d111564e6c/.tmp/A/c39857ed695a4617a2e2053dc79bbc08 2023-04-15 02:54:32,956 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(1227): Checking to see if procedure is done pid=24 2023-04-15 02:54:33,267 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=6.71 KB at sequenceid=1140 (bloomFilter=true), to=hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/b7b06ed62cbe8eca9103c3d111564e6c/.tmp/B/4c557f57c65c4eebb51c4d5b25cf5faa 2023-04-15 02:54:33,687 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=6.71 KB at sequenceid=1140 (bloomFilter=true), to=hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/b7b06ed62cbe8eca9103c3d111564e6c/.tmp/C/5eefa0ef2c104f578fe93f96bd46a3c8 2023-04-15 02:54:33,699 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/b7b06ed62cbe8eca9103c3d111564e6c/.tmp/A/c39857ed695a4617a2e2053dc79bbc08 as hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/b7b06ed62cbe8eca9103c3d111564e6c/A/c39857ed695a4617a2e2053dc79bbc08 2023-04-15 02:54:33,702 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/b7b06ed62cbe8eca9103c3d111564e6c/A/c39857ed695a4617a2e2053dc79bbc08, entries=150, sequenceid=1140, filesize=25.4 K 2023-04-15 02:54:33,703 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/b7b06ed62cbe8eca9103c3d111564e6c/.tmp/B/4c557f57c65c4eebb51c4d5b25cf5faa as hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/b7b06ed62cbe8eca9103c3d111564e6c/B/4c557f57c65c4eebb51c4d5b25cf5faa 2023-04-15 02:54:33,707 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/b7b06ed62cbe8eca9103c3d111564e6c/B/4c557f57c65c4eebb51c4d5b25cf5faa, entries=150, sequenceid=1140, filesize=11.9 K 2023-04-15 02:54:33,707 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/b7b06ed62cbe8eca9103c3d111564e6c/.tmp/C/5eefa0ef2c104f578fe93f96bd46a3c8 as hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/b7b06ed62cbe8eca9103c3d111564e6c/C/5eefa0ef2c104f578fe93f96bd46a3c8 2023-04-15 02:54:33,711 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/b7b06ed62cbe8eca9103c3d111564e6c/C/5eefa0ef2c104f578fe93f96bd46a3c8, entries=150, sequenceid=1140, filesize=11.9 K 2023-04-15 02:54:33,711 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(2948): Finished flush of dataSize ~20.13 KB/20610, heapSize ~60.47 KB/61920, currentSize=0 B/0 for b7b06ed62cbe8eca9103c3d111564e6c in 1704ms, sequenceid=1140, compaction requested=false 2023-04-15 02:54:33,728 DEBUG [StoreCloser-TestAcidGuarantees,,1681527240182.b7b06ed62cbe8eca9103c3d111564e6c.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/b7b06ed62cbe8eca9103c3d111564e6c/A/fc91b7d3afe3454099e64a7900c7f638, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/b7b06ed62cbe8eca9103c3d111564e6c/A/8f538e98aa65491ebfab8d01af624c27, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/b7b06ed62cbe8eca9103c3d111564e6c/A/7219c23fbaee4d32b1f84582e92030c0, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/b7b06ed62cbe8eca9103c3d111564e6c/A/891f9e563f9b42f4aa0368a0f6313a84, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/b7b06ed62cbe8eca9103c3d111564e6c/A/add8b3c6156c490a94ee447b44124c71, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/b7b06ed62cbe8eca9103c3d111564e6c/A/edf1440b49904cc88b8ed1f7e7cb63d4, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/b7b06ed62cbe8eca9103c3d111564e6c/A/5df9409e91d84edc91bc186464378c47, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/b7b06ed62cbe8eca9103c3d111564e6c/A/153faaeb5918489c8f014d09aeb0238e, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/b7b06ed62cbe8eca9103c3d111564e6c/A/a9fb2169d5184cbe8e4682c53928ba72, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/b7b06ed62cbe8eca9103c3d111564e6c/A/4f734e0147ea4a368d6ec7845146f73b, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/b7b06ed62cbe8eca9103c3d111564e6c/A/f062d2172d864cc7aac2415982fffc57, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/b7b06ed62cbe8eca9103c3d111564e6c/A/f789c3b8ecdb49d6ac6ba9ca495a5b11, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/b7b06ed62cbe8eca9103c3d111564e6c/A/6bd682223d1542898d764194cf4d3380, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/b7b06ed62cbe8eca9103c3d111564e6c/A/acdbc16f0bd4470e853d39d9237b4be9, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/b7b06ed62cbe8eca9103c3d111564e6c/A/b745ac63855c4da3bdb4f690d537cf6a, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/b7b06ed62cbe8eca9103c3d111564e6c/A/e8204d9538ec45a184fb961fe5527933, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/b7b06ed62cbe8eca9103c3d111564e6c/A/dcda3bbd8a00469a96653f91fa83ddad, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/b7b06ed62cbe8eca9103c3d111564e6c/A/2b87e81781d04ac4aebf8b72abb33b61, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/b7b06ed62cbe8eca9103c3d111564e6c/A/c591c2fdcd9b445bbac205ab6d1faad6, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/b7b06ed62cbe8eca9103c3d111564e6c/A/efc42ad066a04d8da406d201b8de5f21, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/b7b06ed62cbe8eca9103c3d111564e6c/A/a0b41bd1140847b3b7ae9856b322d46f, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/b7b06ed62cbe8eca9103c3d111564e6c/A/a884776d4eca436ea80b915c5481730c, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/b7b06ed62cbe8eca9103c3d111564e6c/A/5f60f982270942989c1ef49b647c6b9f, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/b7b06ed62cbe8eca9103c3d111564e6c/A/1ddb9af0a434478a984b5c42a480ca02] to archive 2023-04-15 02:54:33,729 DEBUG [StoreCloser-TestAcidGuarantees,,1681527240182.b7b06ed62cbe8eca9103c3d111564e6c.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-15 02:54:33,730 DEBUG [StoreCloser-TestAcidGuarantees,,1681527240182.b7b06ed62cbe8eca9103c3d111564e6c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/b7b06ed62cbe8eca9103c3d111564e6c/A/fc91b7d3afe3454099e64a7900c7f638 to hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/archive/data/default/TestAcidGuarantees/b7b06ed62cbe8eca9103c3d111564e6c/A/fc91b7d3afe3454099e64a7900c7f638 2023-04-15 02:54:33,731 DEBUG [StoreCloser-TestAcidGuarantees,,1681527240182.b7b06ed62cbe8eca9103c3d111564e6c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/b7b06ed62cbe8eca9103c3d111564e6c/A/8f538e98aa65491ebfab8d01af624c27 to hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/archive/data/default/TestAcidGuarantees/b7b06ed62cbe8eca9103c3d111564e6c/A/8f538e98aa65491ebfab8d01af624c27 2023-04-15 02:54:33,732 DEBUG [StoreCloser-TestAcidGuarantees,,1681527240182.b7b06ed62cbe8eca9103c3d111564e6c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/b7b06ed62cbe8eca9103c3d111564e6c/A/7219c23fbaee4d32b1f84582e92030c0 to hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/archive/data/default/TestAcidGuarantees/b7b06ed62cbe8eca9103c3d111564e6c/A/7219c23fbaee4d32b1f84582e92030c0 2023-04-15 02:54:33,733 DEBUG [StoreCloser-TestAcidGuarantees,,1681527240182.b7b06ed62cbe8eca9103c3d111564e6c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/b7b06ed62cbe8eca9103c3d111564e6c/A/891f9e563f9b42f4aa0368a0f6313a84 to hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/archive/data/default/TestAcidGuarantees/b7b06ed62cbe8eca9103c3d111564e6c/A/891f9e563f9b42f4aa0368a0f6313a84 2023-04-15 02:54:33,734 DEBUG [StoreCloser-TestAcidGuarantees,,1681527240182.b7b06ed62cbe8eca9103c3d111564e6c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/b7b06ed62cbe8eca9103c3d111564e6c/A/add8b3c6156c490a94ee447b44124c71 to hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/archive/data/default/TestAcidGuarantees/b7b06ed62cbe8eca9103c3d111564e6c/A/add8b3c6156c490a94ee447b44124c71 2023-04-15 02:54:33,736 DEBUG [StoreCloser-TestAcidGuarantees,,1681527240182.b7b06ed62cbe8eca9103c3d111564e6c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/b7b06ed62cbe8eca9103c3d111564e6c/A/edf1440b49904cc88b8ed1f7e7cb63d4 to hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/archive/data/default/TestAcidGuarantees/b7b06ed62cbe8eca9103c3d111564e6c/A/edf1440b49904cc88b8ed1f7e7cb63d4 2023-04-15 02:54:33,737 DEBUG [StoreCloser-TestAcidGuarantees,,1681527240182.b7b06ed62cbe8eca9103c3d111564e6c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/b7b06ed62cbe8eca9103c3d111564e6c/A/5df9409e91d84edc91bc186464378c47 to hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/archive/data/default/TestAcidGuarantees/b7b06ed62cbe8eca9103c3d111564e6c/A/5df9409e91d84edc91bc186464378c47 2023-04-15 02:54:33,738 DEBUG [StoreCloser-TestAcidGuarantees,,1681527240182.b7b06ed62cbe8eca9103c3d111564e6c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/b7b06ed62cbe8eca9103c3d111564e6c/A/153faaeb5918489c8f014d09aeb0238e to hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/archive/data/default/TestAcidGuarantees/b7b06ed62cbe8eca9103c3d111564e6c/A/153faaeb5918489c8f014d09aeb0238e 2023-04-15 02:54:33,739 DEBUG [StoreCloser-TestAcidGuarantees,,1681527240182.b7b06ed62cbe8eca9103c3d111564e6c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/b7b06ed62cbe8eca9103c3d111564e6c/A/a9fb2169d5184cbe8e4682c53928ba72 to hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/archive/data/default/TestAcidGuarantees/b7b06ed62cbe8eca9103c3d111564e6c/A/a9fb2169d5184cbe8e4682c53928ba72 2023-04-15 02:54:33,740 DEBUG [StoreCloser-TestAcidGuarantees,,1681527240182.b7b06ed62cbe8eca9103c3d111564e6c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/b7b06ed62cbe8eca9103c3d111564e6c/A/4f734e0147ea4a368d6ec7845146f73b to hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/archive/data/default/TestAcidGuarantees/b7b06ed62cbe8eca9103c3d111564e6c/A/4f734e0147ea4a368d6ec7845146f73b 2023-04-15 02:54:33,741 DEBUG [StoreCloser-TestAcidGuarantees,,1681527240182.b7b06ed62cbe8eca9103c3d111564e6c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/b7b06ed62cbe8eca9103c3d111564e6c/A/f062d2172d864cc7aac2415982fffc57 to hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/archive/data/default/TestAcidGuarantees/b7b06ed62cbe8eca9103c3d111564e6c/A/f062d2172d864cc7aac2415982fffc57 2023-04-15 02:54:33,742 DEBUG [StoreCloser-TestAcidGuarantees,,1681527240182.b7b06ed62cbe8eca9103c3d111564e6c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/b7b06ed62cbe8eca9103c3d111564e6c/A/f789c3b8ecdb49d6ac6ba9ca495a5b11 to hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/archive/data/default/TestAcidGuarantees/b7b06ed62cbe8eca9103c3d111564e6c/A/f789c3b8ecdb49d6ac6ba9ca495a5b11 2023-04-15 02:54:33,742 DEBUG [StoreCloser-TestAcidGuarantees,,1681527240182.b7b06ed62cbe8eca9103c3d111564e6c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/b7b06ed62cbe8eca9103c3d111564e6c/A/6bd682223d1542898d764194cf4d3380 to hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/archive/data/default/TestAcidGuarantees/b7b06ed62cbe8eca9103c3d111564e6c/A/6bd682223d1542898d764194cf4d3380 2023-04-15 02:54:33,743 DEBUG [StoreCloser-TestAcidGuarantees,,1681527240182.b7b06ed62cbe8eca9103c3d111564e6c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/b7b06ed62cbe8eca9103c3d111564e6c/A/acdbc16f0bd4470e853d39d9237b4be9 to hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/archive/data/default/TestAcidGuarantees/b7b06ed62cbe8eca9103c3d111564e6c/A/acdbc16f0bd4470e853d39d9237b4be9 2023-04-15 02:54:33,744 DEBUG [StoreCloser-TestAcidGuarantees,,1681527240182.b7b06ed62cbe8eca9103c3d111564e6c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/b7b06ed62cbe8eca9103c3d111564e6c/A/b745ac63855c4da3bdb4f690d537cf6a to hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/archive/data/default/TestAcidGuarantees/b7b06ed62cbe8eca9103c3d111564e6c/A/b745ac63855c4da3bdb4f690d537cf6a 2023-04-15 02:54:33,745 DEBUG [StoreCloser-TestAcidGuarantees,,1681527240182.b7b06ed62cbe8eca9103c3d111564e6c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/b7b06ed62cbe8eca9103c3d111564e6c/A/e8204d9538ec45a184fb961fe5527933 to hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/archive/data/default/TestAcidGuarantees/b7b06ed62cbe8eca9103c3d111564e6c/A/e8204d9538ec45a184fb961fe5527933 2023-04-15 02:54:33,746 DEBUG [StoreCloser-TestAcidGuarantees,,1681527240182.b7b06ed62cbe8eca9103c3d111564e6c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/b7b06ed62cbe8eca9103c3d111564e6c/A/dcda3bbd8a00469a96653f91fa83ddad to hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/archive/data/default/TestAcidGuarantees/b7b06ed62cbe8eca9103c3d111564e6c/A/dcda3bbd8a00469a96653f91fa83ddad 2023-04-15 02:54:33,747 DEBUG [StoreCloser-TestAcidGuarantees,,1681527240182.b7b06ed62cbe8eca9103c3d111564e6c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/b7b06ed62cbe8eca9103c3d111564e6c/A/2b87e81781d04ac4aebf8b72abb33b61 to hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/archive/data/default/TestAcidGuarantees/b7b06ed62cbe8eca9103c3d111564e6c/A/2b87e81781d04ac4aebf8b72abb33b61 2023-04-15 02:54:33,749 DEBUG [StoreCloser-TestAcidGuarantees,,1681527240182.b7b06ed62cbe8eca9103c3d111564e6c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/b7b06ed62cbe8eca9103c3d111564e6c/A/c591c2fdcd9b445bbac205ab6d1faad6 to hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/archive/data/default/TestAcidGuarantees/b7b06ed62cbe8eca9103c3d111564e6c/A/c591c2fdcd9b445bbac205ab6d1faad6 2023-04-15 02:54:33,750 DEBUG [StoreCloser-TestAcidGuarantees,,1681527240182.b7b06ed62cbe8eca9103c3d111564e6c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/b7b06ed62cbe8eca9103c3d111564e6c/A/efc42ad066a04d8da406d201b8de5f21 to hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/archive/data/default/TestAcidGuarantees/b7b06ed62cbe8eca9103c3d111564e6c/A/efc42ad066a04d8da406d201b8de5f21 2023-04-15 02:54:33,751 DEBUG [StoreCloser-TestAcidGuarantees,,1681527240182.b7b06ed62cbe8eca9103c3d111564e6c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/b7b06ed62cbe8eca9103c3d111564e6c/A/a0b41bd1140847b3b7ae9856b322d46f to hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/archive/data/default/TestAcidGuarantees/b7b06ed62cbe8eca9103c3d111564e6c/A/a0b41bd1140847b3b7ae9856b322d46f 2023-04-15 02:54:33,753 DEBUG [StoreCloser-TestAcidGuarantees,,1681527240182.b7b06ed62cbe8eca9103c3d111564e6c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/b7b06ed62cbe8eca9103c3d111564e6c/A/a884776d4eca436ea80b915c5481730c to hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/archive/data/default/TestAcidGuarantees/b7b06ed62cbe8eca9103c3d111564e6c/A/a884776d4eca436ea80b915c5481730c 2023-04-15 02:54:33,755 DEBUG [StoreCloser-TestAcidGuarantees,,1681527240182.b7b06ed62cbe8eca9103c3d111564e6c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/b7b06ed62cbe8eca9103c3d111564e6c/A/5f60f982270942989c1ef49b647c6b9f to hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/archive/data/default/TestAcidGuarantees/b7b06ed62cbe8eca9103c3d111564e6c/A/5f60f982270942989c1ef49b647c6b9f 2023-04-15 02:54:33,757 DEBUG [StoreCloser-TestAcidGuarantees,,1681527240182.b7b06ed62cbe8eca9103c3d111564e6c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/b7b06ed62cbe8eca9103c3d111564e6c/A/1ddb9af0a434478a984b5c42a480ca02 to hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/archive/data/default/TestAcidGuarantees/b7b06ed62cbe8eca9103c3d111564e6c/A/1ddb9af0a434478a984b5c42a480ca02 2023-04-15 02:54:33,760 DEBUG [StoreCloser-TestAcidGuarantees,,1681527240182.b7b06ed62cbe8eca9103c3d111564e6c.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/b7b06ed62cbe8eca9103c3d111564e6c/B/1898232003dd448791f6d6b222731e5a, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/b7b06ed62cbe8eca9103c3d111564e6c/B/91193976fe4c4e15b19485e66f2f6950, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/b7b06ed62cbe8eca9103c3d111564e6c/B/9de6a2bcd583484f8cb8a66f0b62cdbe, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/b7b06ed62cbe8eca9103c3d111564e6c/B/8044099aed62448096cf6b59026c51f9, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/b7b06ed62cbe8eca9103c3d111564e6c/B/21086d2253f94dcea031b131bdd490f8, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/b7b06ed62cbe8eca9103c3d111564e6c/B/cfd9ac1c518c4f72ae895b29915a6ea1, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/b7b06ed62cbe8eca9103c3d111564e6c/B/474c82d609a447279075788f36feb259, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/b7b06ed62cbe8eca9103c3d111564e6c/B/b72caa2df2b34bf69d99434a82ee9ac0, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/b7b06ed62cbe8eca9103c3d111564e6c/B/f562373783fb4d908240d252b4e6387a, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/b7b06ed62cbe8eca9103c3d111564e6c/B/11454a78c616407d849c8a14e7d1674a, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/b7b06ed62cbe8eca9103c3d111564e6c/B/e63856d5116b41328e79561838e6068c, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/b7b06ed62cbe8eca9103c3d111564e6c/B/9b76848f28af4f69ae6328d74ae603f8, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/b7b06ed62cbe8eca9103c3d111564e6c/B/7e797ffd65c5478e80443ace7c715fb7, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/b7b06ed62cbe8eca9103c3d111564e6c/B/860fa6f1f1514f058db8f8453e5c1fc7, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/b7b06ed62cbe8eca9103c3d111564e6c/B/9168c283c4e345d9b692f4ad877325c0, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/b7b06ed62cbe8eca9103c3d111564e6c/B/4813d52cc442437d876f752f5e126c2a, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/b7b06ed62cbe8eca9103c3d111564e6c/B/51687ca44ed04e20b7384f9f2e53d0da, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/b7b06ed62cbe8eca9103c3d111564e6c/B/cd61e6be1fc947dcaee243656edb94cc, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/b7b06ed62cbe8eca9103c3d111564e6c/B/1bb4b536f6f24d5f829bd2d669311137, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/b7b06ed62cbe8eca9103c3d111564e6c/B/6775e098b8ff4b198521a7e961476a2b, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/b7b06ed62cbe8eca9103c3d111564e6c/B/c87760f24be14299b598da8c0f5ec6f5, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/b7b06ed62cbe8eca9103c3d111564e6c/B/c52236b4ba9443eeba5bd3afd5bb2bc6, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/b7b06ed62cbe8eca9103c3d111564e6c/B/d865f93b64654bee8122523c963aa77b, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/b7b06ed62cbe8eca9103c3d111564e6c/B/42954d1787874cf9bb929e1471e57d4d] to archive 2023-04-15 02:54:33,760 DEBUG [StoreCloser-TestAcidGuarantees,,1681527240182.b7b06ed62cbe8eca9103c3d111564e6c.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-15 02:54:33,762 DEBUG [StoreCloser-TestAcidGuarantees,,1681527240182.b7b06ed62cbe8eca9103c3d111564e6c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/b7b06ed62cbe8eca9103c3d111564e6c/B/1898232003dd448791f6d6b222731e5a to hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/archive/data/default/TestAcidGuarantees/b7b06ed62cbe8eca9103c3d111564e6c/B/1898232003dd448791f6d6b222731e5a 2023-04-15 02:54:33,763 DEBUG [StoreCloser-TestAcidGuarantees,,1681527240182.b7b06ed62cbe8eca9103c3d111564e6c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/b7b06ed62cbe8eca9103c3d111564e6c/B/91193976fe4c4e15b19485e66f2f6950 to hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/archive/data/default/TestAcidGuarantees/b7b06ed62cbe8eca9103c3d111564e6c/B/91193976fe4c4e15b19485e66f2f6950 2023-04-15 02:54:33,764 DEBUG [StoreCloser-TestAcidGuarantees,,1681527240182.b7b06ed62cbe8eca9103c3d111564e6c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/b7b06ed62cbe8eca9103c3d111564e6c/B/9de6a2bcd583484f8cb8a66f0b62cdbe to hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/archive/data/default/TestAcidGuarantees/b7b06ed62cbe8eca9103c3d111564e6c/B/9de6a2bcd583484f8cb8a66f0b62cdbe 2023-04-15 02:54:33,765 DEBUG [StoreCloser-TestAcidGuarantees,,1681527240182.b7b06ed62cbe8eca9103c3d111564e6c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/b7b06ed62cbe8eca9103c3d111564e6c/B/8044099aed62448096cf6b59026c51f9 to hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/archive/data/default/TestAcidGuarantees/b7b06ed62cbe8eca9103c3d111564e6c/B/8044099aed62448096cf6b59026c51f9 2023-04-15 02:54:33,766 DEBUG [StoreCloser-TestAcidGuarantees,,1681527240182.b7b06ed62cbe8eca9103c3d111564e6c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/b7b06ed62cbe8eca9103c3d111564e6c/B/21086d2253f94dcea031b131bdd490f8 to hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/archive/data/default/TestAcidGuarantees/b7b06ed62cbe8eca9103c3d111564e6c/B/21086d2253f94dcea031b131bdd490f8 2023-04-15 02:54:33,767 DEBUG [StoreCloser-TestAcidGuarantees,,1681527240182.b7b06ed62cbe8eca9103c3d111564e6c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/b7b06ed62cbe8eca9103c3d111564e6c/B/cfd9ac1c518c4f72ae895b29915a6ea1 to hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/archive/data/default/TestAcidGuarantees/b7b06ed62cbe8eca9103c3d111564e6c/B/cfd9ac1c518c4f72ae895b29915a6ea1 2023-04-15 02:54:33,767 DEBUG [StoreCloser-TestAcidGuarantees,,1681527240182.b7b06ed62cbe8eca9103c3d111564e6c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/b7b06ed62cbe8eca9103c3d111564e6c/B/474c82d609a447279075788f36feb259 to hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/archive/data/default/TestAcidGuarantees/b7b06ed62cbe8eca9103c3d111564e6c/B/474c82d609a447279075788f36feb259 2023-04-15 02:54:33,768 DEBUG [StoreCloser-TestAcidGuarantees,,1681527240182.b7b06ed62cbe8eca9103c3d111564e6c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/b7b06ed62cbe8eca9103c3d111564e6c/B/b72caa2df2b34bf69d99434a82ee9ac0 to hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/archive/data/default/TestAcidGuarantees/b7b06ed62cbe8eca9103c3d111564e6c/B/b72caa2df2b34bf69d99434a82ee9ac0 2023-04-15 02:54:33,769 DEBUG [StoreCloser-TestAcidGuarantees,,1681527240182.b7b06ed62cbe8eca9103c3d111564e6c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/b7b06ed62cbe8eca9103c3d111564e6c/B/f562373783fb4d908240d252b4e6387a to hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/archive/data/default/TestAcidGuarantees/b7b06ed62cbe8eca9103c3d111564e6c/B/f562373783fb4d908240d252b4e6387a 2023-04-15 02:54:33,770 DEBUG [StoreCloser-TestAcidGuarantees,,1681527240182.b7b06ed62cbe8eca9103c3d111564e6c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/b7b06ed62cbe8eca9103c3d111564e6c/B/11454a78c616407d849c8a14e7d1674a to hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/archive/data/default/TestAcidGuarantees/b7b06ed62cbe8eca9103c3d111564e6c/B/11454a78c616407d849c8a14e7d1674a 2023-04-15 02:54:33,771 DEBUG [StoreCloser-TestAcidGuarantees,,1681527240182.b7b06ed62cbe8eca9103c3d111564e6c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/b7b06ed62cbe8eca9103c3d111564e6c/B/e63856d5116b41328e79561838e6068c to hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/archive/data/default/TestAcidGuarantees/b7b06ed62cbe8eca9103c3d111564e6c/B/e63856d5116b41328e79561838e6068c 2023-04-15 02:54:33,772 DEBUG [StoreCloser-TestAcidGuarantees,,1681527240182.b7b06ed62cbe8eca9103c3d111564e6c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/b7b06ed62cbe8eca9103c3d111564e6c/B/9b76848f28af4f69ae6328d74ae603f8 to hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/archive/data/default/TestAcidGuarantees/b7b06ed62cbe8eca9103c3d111564e6c/B/9b76848f28af4f69ae6328d74ae603f8 2023-04-15 02:54:33,773 DEBUG [StoreCloser-TestAcidGuarantees,,1681527240182.b7b06ed62cbe8eca9103c3d111564e6c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/b7b06ed62cbe8eca9103c3d111564e6c/B/7e797ffd65c5478e80443ace7c715fb7 to hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/archive/data/default/TestAcidGuarantees/b7b06ed62cbe8eca9103c3d111564e6c/B/7e797ffd65c5478e80443ace7c715fb7 2023-04-15 02:54:33,773 DEBUG [StoreCloser-TestAcidGuarantees,,1681527240182.b7b06ed62cbe8eca9103c3d111564e6c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/b7b06ed62cbe8eca9103c3d111564e6c/B/860fa6f1f1514f058db8f8453e5c1fc7 to hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/archive/data/default/TestAcidGuarantees/b7b06ed62cbe8eca9103c3d111564e6c/B/860fa6f1f1514f058db8f8453e5c1fc7 2023-04-15 02:54:33,774 DEBUG [StoreCloser-TestAcidGuarantees,,1681527240182.b7b06ed62cbe8eca9103c3d111564e6c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/b7b06ed62cbe8eca9103c3d111564e6c/B/9168c283c4e345d9b692f4ad877325c0 to hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/archive/data/default/TestAcidGuarantees/b7b06ed62cbe8eca9103c3d111564e6c/B/9168c283c4e345d9b692f4ad877325c0 2023-04-15 02:54:33,775 DEBUG [StoreCloser-TestAcidGuarantees,,1681527240182.b7b06ed62cbe8eca9103c3d111564e6c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/b7b06ed62cbe8eca9103c3d111564e6c/B/4813d52cc442437d876f752f5e126c2a to hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/archive/data/default/TestAcidGuarantees/b7b06ed62cbe8eca9103c3d111564e6c/B/4813d52cc442437d876f752f5e126c2a 2023-04-15 02:54:33,776 DEBUG [StoreCloser-TestAcidGuarantees,,1681527240182.b7b06ed62cbe8eca9103c3d111564e6c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/b7b06ed62cbe8eca9103c3d111564e6c/B/51687ca44ed04e20b7384f9f2e53d0da to hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/archive/data/default/TestAcidGuarantees/b7b06ed62cbe8eca9103c3d111564e6c/B/51687ca44ed04e20b7384f9f2e53d0da 2023-04-15 02:54:33,777 DEBUG [StoreCloser-TestAcidGuarantees,,1681527240182.b7b06ed62cbe8eca9103c3d111564e6c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/b7b06ed62cbe8eca9103c3d111564e6c/B/cd61e6be1fc947dcaee243656edb94cc to hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/archive/data/default/TestAcidGuarantees/b7b06ed62cbe8eca9103c3d111564e6c/B/cd61e6be1fc947dcaee243656edb94cc 2023-04-15 02:54:33,778 DEBUG [StoreCloser-TestAcidGuarantees,,1681527240182.b7b06ed62cbe8eca9103c3d111564e6c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/b7b06ed62cbe8eca9103c3d111564e6c/B/1bb4b536f6f24d5f829bd2d669311137 to hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/archive/data/default/TestAcidGuarantees/b7b06ed62cbe8eca9103c3d111564e6c/B/1bb4b536f6f24d5f829bd2d669311137 2023-04-15 02:54:33,779 DEBUG [StoreCloser-TestAcidGuarantees,,1681527240182.b7b06ed62cbe8eca9103c3d111564e6c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/b7b06ed62cbe8eca9103c3d111564e6c/B/6775e098b8ff4b198521a7e961476a2b to hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/archive/data/default/TestAcidGuarantees/b7b06ed62cbe8eca9103c3d111564e6c/B/6775e098b8ff4b198521a7e961476a2b 2023-04-15 02:54:33,780 DEBUG [StoreCloser-TestAcidGuarantees,,1681527240182.b7b06ed62cbe8eca9103c3d111564e6c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/b7b06ed62cbe8eca9103c3d111564e6c/B/c87760f24be14299b598da8c0f5ec6f5 to hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/archive/data/default/TestAcidGuarantees/b7b06ed62cbe8eca9103c3d111564e6c/B/c87760f24be14299b598da8c0f5ec6f5 2023-04-15 02:54:33,781 DEBUG [StoreCloser-TestAcidGuarantees,,1681527240182.b7b06ed62cbe8eca9103c3d111564e6c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/b7b06ed62cbe8eca9103c3d111564e6c/B/c52236b4ba9443eeba5bd3afd5bb2bc6 to hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/archive/data/default/TestAcidGuarantees/b7b06ed62cbe8eca9103c3d111564e6c/B/c52236b4ba9443eeba5bd3afd5bb2bc6 2023-04-15 02:54:33,782 DEBUG [StoreCloser-TestAcidGuarantees,,1681527240182.b7b06ed62cbe8eca9103c3d111564e6c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/b7b06ed62cbe8eca9103c3d111564e6c/B/d865f93b64654bee8122523c963aa77b to hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/archive/data/default/TestAcidGuarantees/b7b06ed62cbe8eca9103c3d111564e6c/B/d865f93b64654bee8122523c963aa77b 2023-04-15 02:54:33,783 DEBUG [StoreCloser-TestAcidGuarantees,,1681527240182.b7b06ed62cbe8eca9103c3d111564e6c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/b7b06ed62cbe8eca9103c3d111564e6c/B/42954d1787874cf9bb929e1471e57d4d to hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/archive/data/default/TestAcidGuarantees/b7b06ed62cbe8eca9103c3d111564e6c/B/42954d1787874cf9bb929e1471e57d4d 2023-04-15 02:54:33,786 DEBUG [StoreCloser-TestAcidGuarantees,,1681527240182.b7b06ed62cbe8eca9103c3d111564e6c.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/b7b06ed62cbe8eca9103c3d111564e6c/C/353790da0a4f4819b1625c8eb352666e, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/b7b06ed62cbe8eca9103c3d111564e6c/C/deb3773513d94fcf990a9633a3cfe8f2, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/b7b06ed62cbe8eca9103c3d111564e6c/C/737e570c76d24414ac42ba9b0bd82ec0, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/b7b06ed62cbe8eca9103c3d111564e6c/C/38eaea4cbf6e46cbbe3107d8924ce952, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/b7b06ed62cbe8eca9103c3d111564e6c/C/d664f390a19847bf93af9bc550f95fda, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/b7b06ed62cbe8eca9103c3d111564e6c/C/8b5d68def4f844bcbbbb22de6f29644a, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/b7b06ed62cbe8eca9103c3d111564e6c/C/dccb7202a6a94ef58345640fce8cc84f, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/b7b06ed62cbe8eca9103c3d111564e6c/C/058660d60eb74ece847d2441e5be3182, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/b7b06ed62cbe8eca9103c3d111564e6c/C/a1fdb8330b4f401484a795f4913c211b, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/b7b06ed62cbe8eca9103c3d111564e6c/C/be56c7c9f6194b33b06978f39ebd5f62, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/b7b06ed62cbe8eca9103c3d111564e6c/C/721c2abb2f524608bf5f421c3d3f9656, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/b7b06ed62cbe8eca9103c3d111564e6c/C/cd3f2f0d61b740ebbce4c8787269d3b1, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/b7b06ed62cbe8eca9103c3d111564e6c/C/6630954ef78e4fa2a6911d94d6b1daba, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/b7b06ed62cbe8eca9103c3d111564e6c/C/24768a1fbb7943f48b50e45cb17b549e, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/b7b06ed62cbe8eca9103c3d111564e6c/C/1f3a607057f74152bacf31d561dd5835, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/b7b06ed62cbe8eca9103c3d111564e6c/C/4b3fbff5251742bbb1852d90743ca06f, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/b7b06ed62cbe8eca9103c3d111564e6c/C/c99c8c8578d84adbb4eb965d5c423d59, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/b7b06ed62cbe8eca9103c3d111564e6c/C/71abddb901ed4f5bb7ff4e17b9d0cf66, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/b7b06ed62cbe8eca9103c3d111564e6c/C/6ca060a294064b6084893685e0c90407, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/b7b06ed62cbe8eca9103c3d111564e6c/C/65fa15372bed4cf0b6d1ef5b6d16d9cd, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/b7b06ed62cbe8eca9103c3d111564e6c/C/796ac175dd4b4d289c2b8b30ae651f10, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/b7b06ed62cbe8eca9103c3d111564e6c/C/24432b816dbe47a08e12696638836f51, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/b7b06ed62cbe8eca9103c3d111564e6c/C/99f8c35b970844cda54d78cd5be15a43] to archive 2023-04-15 02:54:33,787 DEBUG [StoreCloser-TestAcidGuarantees,,1681527240182.b7b06ed62cbe8eca9103c3d111564e6c.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-15 02:54:33,789 DEBUG [StoreCloser-TestAcidGuarantees,,1681527240182.b7b06ed62cbe8eca9103c3d111564e6c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/b7b06ed62cbe8eca9103c3d111564e6c/C/353790da0a4f4819b1625c8eb352666e to hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/archive/data/default/TestAcidGuarantees/b7b06ed62cbe8eca9103c3d111564e6c/C/353790da0a4f4819b1625c8eb352666e 2023-04-15 02:54:33,790 DEBUG [StoreCloser-TestAcidGuarantees,,1681527240182.b7b06ed62cbe8eca9103c3d111564e6c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/b7b06ed62cbe8eca9103c3d111564e6c/C/deb3773513d94fcf990a9633a3cfe8f2 to hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/archive/data/default/TestAcidGuarantees/b7b06ed62cbe8eca9103c3d111564e6c/C/deb3773513d94fcf990a9633a3cfe8f2 2023-04-15 02:54:33,791 DEBUG [StoreCloser-TestAcidGuarantees,,1681527240182.b7b06ed62cbe8eca9103c3d111564e6c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/b7b06ed62cbe8eca9103c3d111564e6c/C/737e570c76d24414ac42ba9b0bd82ec0 to hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/archive/data/default/TestAcidGuarantees/b7b06ed62cbe8eca9103c3d111564e6c/C/737e570c76d24414ac42ba9b0bd82ec0 2023-04-15 02:54:33,792 DEBUG [StoreCloser-TestAcidGuarantees,,1681527240182.b7b06ed62cbe8eca9103c3d111564e6c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/b7b06ed62cbe8eca9103c3d111564e6c/C/38eaea4cbf6e46cbbe3107d8924ce952 to hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/archive/data/default/TestAcidGuarantees/b7b06ed62cbe8eca9103c3d111564e6c/C/38eaea4cbf6e46cbbe3107d8924ce952 2023-04-15 02:54:33,793 DEBUG [StoreCloser-TestAcidGuarantees,,1681527240182.b7b06ed62cbe8eca9103c3d111564e6c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/b7b06ed62cbe8eca9103c3d111564e6c/C/d664f390a19847bf93af9bc550f95fda to hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/archive/data/default/TestAcidGuarantees/b7b06ed62cbe8eca9103c3d111564e6c/C/d664f390a19847bf93af9bc550f95fda 2023-04-15 02:54:33,795 DEBUG [StoreCloser-TestAcidGuarantees,,1681527240182.b7b06ed62cbe8eca9103c3d111564e6c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/b7b06ed62cbe8eca9103c3d111564e6c/C/8b5d68def4f844bcbbbb22de6f29644a to hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/archive/data/default/TestAcidGuarantees/b7b06ed62cbe8eca9103c3d111564e6c/C/8b5d68def4f844bcbbbb22de6f29644a 2023-04-15 02:54:33,796 DEBUG [StoreCloser-TestAcidGuarantees,,1681527240182.b7b06ed62cbe8eca9103c3d111564e6c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/b7b06ed62cbe8eca9103c3d111564e6c/C/dccb7202a6a94ef58345640fce8cc84f to hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/archive/data/default/TestAcidGuarantees/b7b06ed62cbe8eca9103c3d111564e6c/C/dccb7202a6a94ef58345640fce8cc84f 2023-04-15 02:54:33,797 DEBUG [StoreCloser-TestAcidGuarantees,,1681527240182.b7b06ed62cbe8eca9103c3d111564e6c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/b7b06ed62cbe8eca9103c3d111564e6c/C/058660d60eb74ece847d2441e5be3182 to hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/archive/data/default/TestAcidGuarantees/b7b06ed62cbe8eca9103c3d111564e6c/C/058660d60eb74ece847d2441e5be3182 2023-04-15 02:54:33,799 DEBUG [StoreCloser-TestAcidGuarantees,,1681527240182.b7b06ed62cbe8eca9103c3d111564e6c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/b7b06ed62cbe8eca9103c3d111564e6c/C/a1fdb8330b4f401484a795f4913c211b to hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/archive/data/default/TestAcidGuarantees/b7b06ed62cbe8eca9103c3d111564e6c/C/a1fdb8330b4f401484a795f4913c211b 2023-04-15 02:54:33,800 DEBUG [StoreCloser-TestAcidGuarantees,,1681527240182.b7b06ed62cbe8eca9103c3d111564e6c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/b7b06ed62cbe8eca9103c3d111564e6c/C/be56c7c9f6194b33b06978f39ebd5f62 to hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/archive/data/default/TestAcidGuarantees/b7b06ed62cbe8eca9103c3d111564e6c/C/be56c7c9f6194b33b06978f39ebd5f62 2023-04-15 02:54:33,801 DEBUG [StoreCloser-TestAcidGuarantees,,1681527240182.b7b06ed62cbe8eca9103c3d111564e6c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/b7b06ed62cbe8eca9103c3d111564e6c/C/721c2abb2f524608bf5f421c3d3f9656 to hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/archive/data/default/TestAcidGuarantees/b7b06ed62cbe8eca9103c3d111564e6c/C/721c2abb2f524608bf5f421c3d3f9656 2023-04-15 02:54:33,802 DEBUG [StoreCloser-TestAcidGuarantees,,1681527240182.b7b06ed62cbe8eca9103c3d111564e6c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/b7b06ed62cbe8eca9103c3d111564e6c/C/cd3f2f0d61b740ebbce4c8787269d3b1 to hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/archive/data/default/TestAcidGuarantees/b7b06ed62cbe8eca9103c3d111564e6c/C/cd3f2f0d61b740ebbce4c8787269d3b1 2023-04-15 02:54:33,804 DEBUG [StoreCloser-TestAcidGuarantees,,1681527240182.b7b06ed62cbe8eca9103c3d111564e6c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/b7b06ed62cbe8eca9103c3d111564e6c/C/6630954ef78e4fa2a6911d94d6b1daba to hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/archive/data/default/TestAcidGuarantees/b7b06ed62cbe8eca9103c3d111564e6c/C/6630954ef78e4fa2a6911d94d6b1daba 2023-04-15 02:54:33,806 DEBUG [StoreCloser-TestAcidGuarantees,,1681527240182.b7b06ed62cbe8eca9103c3d111564e6c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/b7b06ed62cbe8eca9103c3d111564e6c/C/24768a1fbb7943f48b50e45cb17b549e to hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/archive/data/default/TestAcidGuarantees/b7b06ed62cbe8eca9103c3d111564e6c/C/24768a1fbb7943f48b50e45cb17b549e 2023-04-15 02:54:33,807 DEBUG [StoreCloser-TestAcidGuarantees,,1681527240182.b7b06ed62cbe8eca9103c3d111564e6c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/b7b06ed62cbe8eca9103c3d111564e6c/C/1f3a607057f74152bacf31d561dd5835 to hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/archive/data/default/TestAcidGuarantees/b7b06ed62cbe8eca9103c3d111564e6c/C/1f3a607057f74152bacf31d561dd5835 2023-04-15 02:54:33,808 DEBUG [StoreCloser-TestAcidGuarantees,,1681527240182.b7b06ed62cbe8eca9103c3d111564e6c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/b7b06ed62cbe8eca9103c3d111564e6c/C/4b3fbff5251742bbb1852d90743ca06f to hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/archive/data/default/TestAcidGuarantees/b7b06ed62cbe8eca9103c3d111564e6c/C/4b3fbff5251742bbb1852d90743ca06f 2023-04-15 02:54:33,810 DEBUG [StoreCloser-TestAcidGuarantees,,1681527240182.b7b06ed62cbe8eca9103c3d111564e6c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/b7b06ed62cbe8eca9103c3d111564e6c/C/c99c8c8578d84adbb4eb965d5c423d59 to hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/archive/data/default/TestAcidGuarantees/b7b06ed62cbe8eca9103c3d111564e6c/C/c99c8c8578d84adbb4eb965d5c423d59 2023-04-15 02:54:33,811 DEBUG [StoreCloser-TestAcidGuarantees,,1681527240182.b7b06ed62cbe8eca9103c3d111564e6c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/b7b06ed62cbe8eca9103c3d111564e6c/C/71abddb901ed4f5bb7ff4e17b9d0cf66 to hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/archive/data/default/TestAcidGuarantees/b7b06ed62cbe8eca9103c3d111564e6c/C/71abddb901ed4f5bb7ff4e17b9d0cf66 2023-04-15 02:54:33,812 DEBUG [StoreCloser-TestAcidGuarantees,,1681527240182.b7b06ed62cbe8eca9103c3d111564e6c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/b7b06ed62cbe8eca9103c3d111564e6c/C/6ca060a294064b6084893685e0c90407 to hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/archive/data/default/TestAcidGuarantees/b7b06ed62cbe8eca9103c3d111564e6c/C/6ca060a294064b6084893685e0c90407 2023-04-15 02:54:33,813 DEBUG [StoreCloser-TestAcidGuarantees,,1681527240182.b7b06ed62cbe8eca9103c3d111564e6c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/b7b06ed62cbe8eca9103c3d111564e6c/C/65fa15372bed4cf0b6d1ef5b6d16d9cd to hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/archive/data/default/TestAcidGuarantees/b7b06ed62cbe8eca9103c3d111564e6c/C/65fa15372bed4cf0b6d1ef5b6d16d9cd 2023-04-15 02:54:33,814 DEBUG [StoreCloser-TestAcidGuarantees,,1681527240182.b7b06ed62cbe8eca9103c3d111564e6c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/b7b06ed62cbe8eca9103c3d111564e6c/C/796ac175dd4b4d289c2b8b30ae651f10 to hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/archive/data/default/TestAcidGuarantees/b7b06ed62cbe8eca9103c3d111564e6c/C/796ac175dd4b4d289c2b8b30ae651f10 2023-04-15 02:54:33,815 DEBUG [StoreCloser-TestAcidGuarantees,,1681527240182.b7b06ed62cbe8eca9103c3d111564e6c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/b7b06ed62cbe8eca9103c3d111564e6c/C/24432b816dbe47a08e12696638836f51 to hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/archive/data/default/TestAcidGuarantees/b7b06ed62cbe8eca9103c3d111564e6c/C/24432b816dbe47a08e12696638836f51 2023-04-15 02:54:33,816 DEBUG [StoreCloser-TestAcidGuarantees,,1681527240182.b7b06ed62cbe8eca9103c3d111564e6c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/b7b06ed62cbe8eca9103c3d111564e6c/C/99f8c35b970844cda54d78cd5be15a43 to hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/archive/data/default/TestAcidGuarantees/b7b06ed62cbe8eca9103c3d111564e6c/C/99f8c35b970844cda54d78cd5be15a43 2023-04-15 02:54:33,820 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/b7b06ed62cbe8eca9103c3d111564e6c/recovered.edits/1143.seqid, newMaxSeqId=1143, maxSeqId=4 2023-04-15 02:54:33,822 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1681527240182.b7b06ed62cbe8eca9103c3d111564e6c. 2023-04-15 02:54:33,822 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1558): Region close journal for b7b06ed62cbe8eca9103c3d111564e6c: 2023-04-15 02:54:33,824 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] handler.UnassignRegionHandler(147): Closed b7b06ed62cbe8eca9103c3d111564e6c 2023-04-15 02:54:33,824 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=25 updating hbase:meta row=b7b06ed62cbe8eca9103c3d111564e6c, regionState=CLOSED 2023-04-15 02:54:33,824 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1681527240182.b7b06ed62cbe8eca9103c3d111564e6c.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1681527273824"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1681527273824"}]},"ts":"1681527273824"} 2023-04-15 02:54:33,826 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=26, resume processing ppid=25 2023-04-15 02:54:33,826 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=26, ppid=25, state=SUCCESS; CloseRegionProcedure b7b06ed62cbe8eca9103c3d111564e6c, server=jenkins-hbase20.apache.org,34681,1681527209133 in 1.9710 sec 2023-04-15 02:54:33,827 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=25, resume processing ppid=24 2023-04-15 02:54:33,827 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=25, ppid=24, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=b7b06ed62cbe8eca9103c3d111564e6c, UNASSIGN in 1.9740 sec 2023-04-15 02:54:33,828 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1681527273828"}]},"ts":"1681527273828"} 2023-04-15 02:54:33,828 INFO [PEWorker-4] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=DISABLED in hbase:meta 2023-04-15 02:54:33,829 INFO [PEWorker-4] procedure.DisableTableProcedure(305): Set TestAcidGuarantees to state=DISABLED 2023-04-15 02:54:33,830 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=24, state=SUCCESS; DisableTableProcedure table=TestAcidGuarantees in 1.9830 sec 2023-04-15 02:54:33,957 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(1227): Checking to see if procedure is done pid=24 2023-04-15 02:54:33,957 INFO [Listener at localhost.localdomain/33693] client.HBaseAdmin$TableFuture(3541): Operation: DISABLE, Table Name: default:TestAcidGuarantees, procId: 24 completed 2023-04-15 02:54:33,958 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.HMaster$5(2228): Client=jenkins//148.251.75.209 delete TestAcidGuarantees 2023-04-15 02:54:33,959 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] procedure2.ProcedureExecutor(1029): Stored pid=27, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION; DeleteTableProcedure table=TestAcidGuarantees 2023-04-15 02:54:33,960 DEBUG [PEWorker-3] procedure.DeleteTableProcedure(101): Waiting for RIT for pid=27, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-15 02:54:33,961 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(1227): Checking to see if procedure is done pid=27 2023-04-15 02:54:33,961 DEBUG [PEWorker-3] procedure.DeleteTableProcedure(113): Deleting regions from filesystem for pid=27, state=RUNNABLE:DELETE_TABLE_CLEAR_FS_LAYOUT, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-15 02:54:33,965 DEBUG [HFileArchiver-5] backup.HFileArchiver(131): ARCHIVING hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/.tmp/data/default/TestAcidGuarantees/b7b06ed62cbe8eca9103c3d111564e6c 2023-04-15 02:54:33,967 DEBUG [HFileArchiver-5] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/.tmp/data/default/TestAcidGuarantees/b7b06ed62cbe8eca9103c3d111564e6c/A, FileablePath, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/.tmp/data/default/TestAcidGuarantees/b7b06ed62cbe8eca9103c3d111564e6c/B, FileablePath, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/.tmp/data/default/TestAcidGuarantees/b7b06ed62cbe8eca9103c3d111564e6c/C, FileablePath, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/.tmp/data/default/TestAcidGuarantees/b7b06ed62cbe8eca9103c3d111564e6c/recovered.edits] 2023-04-15 02:54:33,969 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/.tmp/data/default/TestAcidGuarantees/b7b06ed62cbe8eca9103c3d111564e6c/A/c39857ed695a4617a2e2053dc79bbc08 to hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/archive/data/default/TestAcidGuarantees/b7b06ed62cbe8eca9103c3d111564e6c/A/c39857ed695a4617a2e2053dc79bbc08 2023-04-15 02:54:33,971 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/.tmp/data/default/TestAcidGuarantees/b7b06ed62cbe8eca9103c3d111564e6c/A/f749db64ccee454aad6728bb945bb767 to hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/archive/data/default/TestAcidGuarantees/b7b06ed62cbe8eca9103c3d111564e6c/A/f749db64ccee454aad6728bb945bb767 2023-04-15 02:54:33,973 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/.tmp/data/default/TestAcidGuarantees/b7b06ed62cbe8eca9103c3d111564e6c/B/4c557f57c65c4eebb51c4d5b25cf5faa to hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/archive/data/default/TestAcidGuarantees/b7b06ed62cbe8eca9103c3d111564e6c/B/4c557f57c65c4eebb51c4d5b25cf5faa 2023-04-15 02:54:33,974 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/.tmp/data/default/TestAcidGuarantees/b7b06ed62cbe8eca9103c3d111564e6c/B/8a2a6e4ea69545cc869d1bc0e05760b1 to hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/archive/data/default/TestAcidGuarantees/b7b06ed62cbe8eca9103c3d111564e6c/B/8a2a6e4ea69545cc869d1bc0e05760b1 2023-04-15 02:54:33,977 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/.tmp/data/default/TestAcidGuarantees/b7b06ed62cbe8eca9103c3d111564e6c/C/5eefa0ef2c104f578fe93f96bd46a3c8 to hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/archive/data/default/TestAcidGuarantees/b7b06ed62cbe8eca9103c3d111564e6c/C/5eefa0ef2c104f578fe93f96bd46a3c8 2023-04-15 02:54:33,978 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/.tmp/data/default/TestAcidGuarantees/b7b06ed62cbe8eca9103c3d111564e6c/C/8be11b0f6a924755abe77e5d979844e6 to hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/archive/data/default/TestAcidGuarantees/b7b06ed62cbe8eca9103c3d111564e6c/C/8be11b0f6a924755abe77e5d979844e6 2023-04-15 02:54:33,981 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/.tmp/data/default/TestAcidGuarantees/b7b06ed62cbe8eca9103c3d111564e6c/recovered.edits/1143.seqid to hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/archive/data/default/TestAcidGuarantees/b7b06ed62cbe8eca9103c3d111564e6c/recovered.edits/1143.seqid 2023-04-15 02:54:33,981 DEBUG [HFileArchiver-5] backup.HFileArchiver(596): Deleted hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/.tmp/data/default/TestAcidGuarantees/b7b06ed62cbe8eca9103c3d111564e6c 2023-04-15 02:54:33,981 DEBUG [PEWorker-3] procedure.DeleteTableProcedure(328): Archived TestAcidGuarantees regions 2023-04-15 02:54:33,982 DEBUG [PEWorker-3] backup.HFileArchiver(131): ARCHIVING hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3 2023-04-15 02:54:33,983 DEBUG [PEWorker-3] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A] 2023-04-15 02:54:33,986 DEBUG [PEWorker-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230415065542fc348c48a28f3acb41347d7764 to hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230415065542fc348c48a28f3acb41347d7764 2023-04-15 02:54:33,987 DEBUG [PEWorker-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023041533969fcf9d9a442996707e7dfe09d3af to hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023041533969fcf9d9a442996707e7dfe09d3af 2023-04-15 02:54:33,988 DEBUG [PEWorker-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023041543f8b8a274024fc9b450f2469cd07fea to hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023041543f8b8a274024fc9b450f2469cd07fea 2023-04-15 02:54:33,989 DEBUG [PEWorker-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230415441230a350ff4baa9f843583b71686f1 to hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230415441230a350ff4baa9f843583b71686f1 2023-04-15 02:54:33,990 DEBUG [PEWorker-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023041545c8ae0cb6d54fdfa28db895d93771e4 to hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023041545c8ae0cb6d54fdfa28db895d93771e4 2023-04-15 02:54:33,991 DEBUG [PEWorker-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304155e45feaadf0847e5888bb142e4851022 to hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304155e45feaadf0847e5888bb142e4851022 2023-04-15 02:54:33,992 DEBUG [PEWorker-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304156304ea2649aa4d86b4ab10dbd3a25d7a to hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304156304ea2649aa4d86b4ab10dbd3a25d7a 2023-04-15 02:54:33,993 DEBUG [PEWorker-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230415641aa4a51611443e8dc71c064d4e6264 to hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230415641aa4a51611443e8dc71c064d4e6264 2023-04-15 02:54:33,994 DEBUG [PEWorker-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304156f45af92fd8b4759a0507d03cb177d33 to hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304156f45af92fd8b4759a0507d03cb177d33 2023-04-15 02:54:33,995 DEBUG [PEWorker-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304157131d7079c734f1097b575fba3f782d0 to hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304157131d7079c734f1097b575fba3f782d0 2023-04-15 02:54:33,996 DEBUG [PEWorker-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230415974936be982245069df25bf634423c28 to hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230415974936be982245069df25bf634423c28 2023-04-15 02:54:33,997 DEBUG [PEWorker-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230415b9a16768821f4f3ebc37383f80930a9e to hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230415b9a16768821f4f3ebc37383f80930a9e 2023-04-15 02:54:33,998 DEBUG [PEWorker-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230415c1c28ef1b35541eba6007526680031cc to hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230415c1c28ef1b35541eba6007526680031cc 2023-04-15 02:54:33,999 DEBUG [PEWorker-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230415c37111470da7478abf1a1fdf2f9829a3 to hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230415c37111470da7478abf1a1fdf2f9829a3 2023-04-15 02:54:33,999 DEBUG [PEWorker-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230415c50d7e5025b7481da1600302c3db1bae to hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230415c50d7e5025b7481da1600302c3db1bae 2023-04-15 02:54:34,000 DEBUG [PEWorker-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230415d29b7707d415438fa4fc4885b9851540 to hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230415d29b7707d415438fa4fc4885b9851540 2023-04-15 02:54:34,001 DEBUG [PEWorker-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230415df42d3450bf14f5097ffe9fc14ba834d to hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230415df42d3450bf14f5097ffe9fc14ba834d 2023-04-15 02:54:34,002 DEBUG [PEWorker-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230415eb6dfbfce2e946da9ce20208b2efeab4 to hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230415eb6dfbfce2e946da9ce20208b2efeab4 2023-04-15 02:54:34,003 DEBUG [PEWorker-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230415fb515c3897bc482a88c85a390a48b21e to hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230415fb515c3897bc482a88c85a390a48b21e 2023-04-15 02:54:34,004 DEBUG [PEWorker-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230415fe29b74e7a024e8198d854a3dca04cd2 to hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230415fe29b74e7a024e8198d854a3dca04cd2 2023-04-15 02:54:34,004 DEBUG [PEWorker-3] backup.HFileArchiver(596): Deleted hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3 2023-04-15 02:54:34,006 DEBUG [PEWorker-3] procedure.DeleteTableProcedure(118): Deleting regions from META for pid=27, state=RUNNABLE:DELETE_TABLE_REMOVE_FROM_META, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-15 02:54:34,008 WARN [PEWorker-3] procedure.DeleteTableProcedure(384): Deleting some vestigial 1 rows of TestAcidGuarantees from hbase:meta 2023-04-15 02:54:34,010 DEBUG [PEWorker-3] procedure.DeleteTableProcedure(421): Removing 'TestAcidGuarantees' descriptor. 2023-04-15 02:54:34,011 DEBUG [PEWorker-3] procedure.DeleteTableProcedure(124): Deleting assignment state for pid=27, state=RUNNABLE:DELETE_TABLE_UNASSIGN_REGIONS, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-15 02:54:34,011 DEBUG [PEWorker-3] procedure.DeleteTableProcedure(411): Removing 'TestAcidGuarantees' from region states. 2023-04-15 02:54:34,011 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"TestAcidGuarantees,,1681527240182.b7b06ed62cbe8eca9103c3d111564e6c.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1681527274011"}]},"ts":"9223372036854775807"} 2023-04-15 02:54:34,013 INFO [PEWorker-3] hbase.MetaTableAccessor(1788): Deleted 1 regions from META 2023-04-15 02:54:34,013 DEBUG [PEWorker-3] hbase.MetaTableAccessor(1789): Deleted regions: [{ENCODED => b7b06ed62cbe8eca9103c3d111564e6c, NAME => 'TestAcidGuarantees,,1681527240182.b7b06ed62cbe8eca9103c3d111564e6c.', STARTKEY => '', ENDKEY => ''}] 2023-04-15 02:54:34,013 DEBUG [PEWorker-3] procedure.DeleteTableProcedure(415): Marking 'TestAcidGuarantees' as deleted. 2023-04-15 02:54:34,013 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":0,"tag":[],"timestamp":"1681527274013"}]},"ts":"9223372036854775807"} 2023-04-15 02:54:34,014 INFO [PEWorker-3] hbase.MetaTableAccessor(1658): Deleted table TestAcidGuarantees state from META 2023-04-15 02:54:34,016 DEBUG [PEWorker-3] procedure.DeleteTableProcedure(130): Finished pid=27, state=RUNNABLE:DELETE_TABLE_POST_OPERATION, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-15 02:54:34,017 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=27, state=SUCCESS; DeleteTableProcedure table=TestAcidGuarantees in 58 msec 2023-04-15 02:54:34,061 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(1227): Checking to see if procedure is done pid=27 2023-04-15 02:54:34,062 INFO [Listener at localhost.localdomain/33693] client.HBaseAdmin$TableFuture(3541): Operation: DELETE, Table Name: default:TestAcidGuarantees, procId: 27 completed 2023-04-15 02:54:34,072 INFO [Listener at localhost.localdomain/33693] hbase.ResourceChecker(175): after: TestAcidGuaranteesWithEagerPolicy#testMobMixedAtomicity Thread=224 (was 234), OpenFileDescriptor=517 (was 560), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=373 (was 397), ProcessCount=169 (was 169), AvailableMemoryMB=5720 (was 5901) 2023-04-15 02:54:34,081 INFO [Listener at localhost.localdomain/33693] hbase.ResourceChecker(147): before: TestAcidGuaranteesWithEagerPolicy#testGetAtomicity Thread=224, OpenFileDescriptor=517, MaxFileDescriptor=60000, SystemLoadAverage=373, ProcessCount=169, AvailableMemoryMB=5720 2023-04-15 02:54:34,083 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] util.TableDescriptorChecker(340): MEMSTORE_FLUSHSIZE for table descriptor or "hbase.hregion.memstore.flush.size" (131072) is too small, which might cause very frequent flushing. 2023-04-15 02:54:34,084 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.HMaster$4(2112): Client=jenkins//148.251.75.209 create 'TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-04-15 02:54:34,085 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] procedure2.ProcedureExecutor(1029): Stored pid=28, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION; CreateTableProcedure table=TestAcidGuarantees 2023-04-15 02:54:34,086 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=28, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_PRE_OPERATION 2023-04-15 02:54:34,086 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(697): Client=jenkins//148.251.75.209 procedure request for creating table: namespace: "default" qualifier: "TestAcidGuarantees" procId is: 28 2023-04-15 02:54:34,087 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=28, state=RUNNABLE:CREATE_TABLE_WRITE_FS_LAYOUT, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_WRITE_FS_LAYOUT 2023-04-15 02:54:34,087 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(1227): Checking to see if procedure is done pid=28 2023-04-15 02:54:34,088 DEBUG [HFileArchiver-6] backup.HFileArchiver(131): ARCHIVING hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/.tmp/data/default/TestAcidGuarantees/f8160290180ba126c904acc32b60eac7 2023-04-15 02:54:34,088 DEBUG [HFileArchiver-6] backup.HFileArchiver(153): Directory hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/.tmp/data/default/TestAcidGuarantees/f8160290180ba126c904acc32b60eac7 empty. 2023-04-15 02:54:34,089 DEBUG [HFileArchiver-6] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/.tmp/data/default/TestAcidGuarantees/f8160290180ba126c904acc32b60eac7 2023-04-15 02:54:34,089 DEBUG [PEWorker-2] procedure.DeleteTableProcedure(328): Archived TestAcidGuarantees regions 2023-04-15 02:54:34,189 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(1227): Checking to see if procedure is done pid=28 2023-04-15 02:54:34,390 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(1227): Checking to see if procedure is done pid=28 2023-04-15 02:54:34,500 DEBUG [PEWorker-2] util.FSTableDescriptors(570): Wrote into hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/.tmp/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000001 2023-04-15 02:54:34,505 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(7675): creating {ENCODED => f8160290180ba126c904acc32b60eac7, NAME => 'TestAcidGuarantees,,1681527274083.f8160290180ba126c904acc32b60eac7.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/.tmp 2023-04-15 02:54:34,693 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(1227): Checking to see if procedure is done pid=28 2023-04-15 02:54:34,913 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1681527274083.f8160290180ba126c904acc32b60eac7.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-15 02:54:34,914 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1604): Closing f8160290180ba126c904acc32b60eac7, disabling compactions & flushes 2023-04-15 02:54:34,914 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1681527274083.f8160290180ba126c904acc32b60eac7. 2023-04-15 02:54:34,914 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1681527274083.f8160290180ba126c904acc32b60eac7. 2023-04-15 02:54:34,914 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1681527274083.f8160290180ba126c904acc32b60eac7. after waiting 0 ms 2023-04-15 02:54:34,914 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1681527274083.f8160290180ba126c904acc32b60eac7. 2023-04-15 02:54:34,914 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1681527274083.f8160290180ba126c904acc32b60eac7. 2023-04-15 02:54:34,915 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1558): Region close journal for f8160290180ba126c904acc32b60eac7: 2023-04-15 02:54:34,922 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=28, state=RUNNABLE:CREATE_TABLE_ADD_TO_META, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_ADD_TO_META 2023-04-15 02:54:34,924 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1681527274083.f8160290180ba126c904acc32b60eac7.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1681527274924"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1681527274924"}]},"ts":"1681527274924"} 2023-04-15 02:54:34,926 INFO [PEWorker-2] hbase.MetaTableAccessor(1496): Added 1 regions to meta. 2023-04-15 02:54:34,927 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=28, state=RUNNABLE:CREATE_TABLE_ASSIGN_REGIONS, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_ASSIGN_REGIONS 2023-04-15 02:54:34,927 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1681527274927"}]},"ts":"1681527274927"} 2023-04-15 02:54:34,929 INFO [PEWorker-2] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=ENABLING in hbase:meta 2023-04-15 02:54:34,933 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=29, ppid=28, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=TestAcidGuarantees, region=f8160290180ba126c904acc32b60eac7, ASSIGN}] 2023-04-15 02:54:34,935 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=29, ppid=28, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=TestAcidGuarantees, region=f8160290180ba126c904acc32b60eac7, ASSIGN 2023-04-15 02:54:34,936 INFO [PEWorker-1] assignment.TransitRegionStateProcedure(193): Starting pid=29, ppid=28, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=TestAcidGuarantees, region=f8160290180ba126c904acc32b60eac7, ASSIGN; state=OFFLINE, location=jenkins-hbase20.apache.org,34681,1681527209133; forceNewPlan=false, retain=false 2023-04-15 02:54:35,088 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=29 updating hbase:meta row=f8160290180ba126c904acc32b60eac7, regionState=OPENING, regionLocation=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:35,089 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1681527274083.f8160290180ba126c904acc32b60eac7.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1681527275088"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1681527275088"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1681527275088"}]},"ts":"1681527275088"} 2023-04-15 02:54:35,092 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=30, ppid=29, state=RUNNABLE; OpenRegionProcedure f8160290180ba126c904acc32b60eac7, server=jenkins-hbase20.apache.org,34681,1681527209133}] 2023-04-15 02:54:35,195 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(1227): Checking to see if procedure is done pid=28 2023-04-15 02:54:35,259 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] handler.AssignRegionHandler(128): Open TestAcidGuarantees,,1681527274083.f8160290180ba126c904acc32b60eac7. 2023-04-15 02:54:35,259 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => f8160290180ba126c904acc32b60eac7, NAME => 'TestAcidGuarantees,,1681527274083.f8160290180ba126c904acc32b60eac7.', STARTKEY => '', ENDKEY => ''} 2023-04-15 02:54:35,260 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table TestAcidGuarantees f8160290180ba126c904acc32b60eac7 2023-04-15 02:54:35,260 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1681527274083.f8160290180ba126c904acc32b60eac7.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-15 02:54:35,260 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7894): checking encryption for f8160290180ba126c904acc32b60eac7 2023-04-15 02:54:35,260 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7897): checking classloading for f8160290180ba126c904acc32b60eac7 2023-04-15 02:54:35,263 INFO [StoreOpener-f8160290180ba126c904acc32b60eac7-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family A of region f8160290180ba126c904acc32b60eac7 2023-04-15 02:54:35,266 DEBUG [StoreOpener-f8160290180ba126c904acc32b60eac7-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/f8160290180ba126c904acc32b60eac7/A 2023-04-15 02:54:35,266 DEBUG [StoreOpener-f8160290180ba126c904acc32b60eac7-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/f8160290180ba126c904acc32b60eac7/A 2023-04-15 02:54:35,267 INFO [StoreOpener-f8160290180ba126c904acc32b60eac7-1] regionserver.CompactingMemStore(121): Store=A, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-15 02:54:35,267 INFO [StoreOpener-f8160290180ba126c904acc32b60eac7-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region f8160290180ba126c904acc32b60eac7 columnFamilyName A 2023-04-15 02:54:35,268 INFO [StoreOpener-f8160290180ba126c904acc32b60eac7-1] regionserver.HStore(310): Store=f8160290180ba126c904acc32b60eac7/A, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-15 02:54:35,268 INFO [StoreOpener-f8160290180ba126c904acc32b60eac7-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family B of region f8160290180ba126c904acc32b60eac7 2023-04-15 02:54:35,270 DEBUG [StoreOpener-f8160290180ba126c904acc32b60eac7-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/f8160290180ba126c904acc32b60eac7/B 2023-04-15 02:54:35,271 DEBUG [StoreOpener-f8160290180ba126c904acc32b60eac7-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/f8160290180ba126c904acc32b60eac7/B 2023-04-15 02:54:35,271 INFO [StoreOpener-f8160290180ba126c904acc32b60eac7-1] regionserver.CompactingMemStore(121): Store=B, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-15 02:54:35,271 INFO [StoreOpener-f8160290180ba126c904acc32b60eac7-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region f8160290180ba126c904acc32b60eac7 columnFamilyName B 2023-04-15 02:54:35,272 INFO [StoreOpener-f8160290180ba126c904acc32b60eac7-1] regionserver.HStore(310): Store=f8160290180ba126c904acc32b60eac7/B, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-15 02:54:35,272 INFO [StoreOpener-f8160290180ba126c904acc32b60eac7-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family C of region f8160290180ba126c904acc32b60eac7 2023-04-15 02:54:35,274 DEBUG [StoreOpener-f8160290180ba126c904acc32b60eac7-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/f8160290180ba126c904acc32b60eac7/C 2023-04-15 02:54:35,274 DEBUG [StoreOpener-f8160290180ba126c904acc32b60eac7-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/f8160290180ba126c904acc32b60eac7/C 2023-04-15 02:54:35,274 INFO [StoreOpener-f8160290180ba126c904acc32b60eac7-1] regionserver.CompactingMemStore(121): Store=C, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-15 02:54:35,274 INFO [StoreOpener-f8160290180ba126c904acc32b60eac7-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region f8160290180ba126c904acc32b60eac7 columnFamilyName C 2023-04-15 02:54:35,275 INFO [StoreOpener-f8160290180ba126c904acc32b60eac7-1] regionserver.HStore(310): Store=f8160290180ba126c904acc32b60eac7/C, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-15 02:54:35,275 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1146): Setting FlushNonSloppyStoresFirstPolicy for the region=TestAcidGuarantees,,1681527274083.f8160290180ba126c904acc32b60eac7. 2023-04-15 02:54:35,276 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/f8160290180ba126c904acc32b60eac7 2023-04-15 02:54:35,277 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/f8160290180ba126c904acc32b60eac7 2023-04-15 02:54:35,281 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table TestAcidGuarantees descriptor;using region.getMemStoreFlushHeapSize/# of families (16.0 M)) instead. 2023-04-15 02:54:35,282 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1055): writing seq id for f8160290180ba126c904acc32b60eac7 2023-04-15 02:54:35,286 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/f8160290180ba126c904acc32b60eac7/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-04-15 02:54:35,286 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1072): Opened f8160290180ba126c904acc32b60eac7; next sequenceid=2; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=63856349, jitterRate=-0.04846625030040741}, FlushLargeStoresPolicy{flushSizeLowerBound=16777216} 2023-04-15 02:54:35,286 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(965): Region open journal for f8160290180ba126c904acc32b60eac7: 2023-04-15 02:54:35,287 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for TestAcidGuarantees,,1681527274083.f8160290180ba126c904acc32b60eac7., pid=30, masterSystemTime=1681527275247 2023-04-15 02:54:35,288 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for TestAcidGuarantees,,1681527274083.f8160290180ba126c904acc32b60eac7. 2023-04-15 02:54:35,288 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] handler.AssignRegionHandler(156): Opened TestAcidGuarantees,,1681527274083.f8160290180ba126c904acc32b60eac7. 2023-04-15 02:54:35,289 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=29 updating hbase:meta row=f8160290180ba126c904acc32b60eac7, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:35,289 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"TestAcidGuarantees,,1681527274083.f8160290180ba126c904acc32b60eac7.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1681527275288"},{"qualifier":"server","vlen":32,"tag":[],"timestamp":"1681527275288"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1681527275288"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1681527275288"}]},"ts":"1681527275288"} 2023-04-15 02:54:35,291 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=30, resume processing ppid=29 2023-04-15 02:54:35,291 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=30, ppid=29, state=SUCCESS; OpenRegionProcedure f8160290180ba126c904acc32b60eac7, server=jenkins-hbase20.apache.org,34681,1681527209133 in 198 msec 2023-04-15 02:54:35,293 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=29, resume processing ppid=28 2023-04-15 02:54:35,293 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=29, ppid=28, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=f8160290180ba126c904acc32b60eac7, ASSIGN in 358 msec 2023-04-15 02:54:35,293 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=28, state=RUNNABLE:CREATE_TABLE_UPDATE_DESC_CACHE, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_UPDATE_DESC_CACHE 2023-04-15 02:54:35,294 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1681527275294"}]},"ts":"1681527275294"} 2023-04-15 02:54:35,295 INFO [PEWorker-1] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=ENABLED in hbase:meta 2023-04-15 02:54:35,296 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=28, state=RUNNABLE:CREATE_TABLE_POST_OPERATION, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_POST_OPERATION 2023-04-15 02:54:35,298 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=28, state=SUCCESS; CreateTableProcedure table=TestAcidGuarantees in 1.2120 sec 2023-04-15 02:54:36,197 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(1227): Checking to see if procedure is done pid=28 2023-04-15 02:54:36,198 INFO [Listener at localhost.localdomain/33693] client.HBaseAdmin$TableFuture(3541): Operation: CREATE, Table Name: default:TestAcidGuarantees, procId: 28 completed 2023-04-15 02:54:36,199 DEBUG [Listener at localhost.localdomain/33693] zookeeper.ReadOnlyZKClient(139): Connect 0x29efb2d8 to 127.0.0.1:51762 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-15 02:54:36,203 DEBUG [Listener at localhost.localdomain/33693] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@16367fdc, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-15 02:54:36,204 DEBUG [Listener at localhost.localdomain/33693] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-15 02:54:36,205 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:41878, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-15 02:54:36,206 DEBUG [Listener at localhost.localdomain/33693] ipc.RpcConnection(124): Using SIMPLE authentication for service=MasterService, sasl=false 2023-04-15 02:54:36,208 INFO [RS-EventLoopGroup-1-3] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:44552, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=MasterService 2023-04-15 02:54:36,210 DEBUG [Listener at localhost.localdomain/33693] zookeeper.ReadOnlyZKClient(139): Connect 0x64e0df67 to 127.0.0.1:51762 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-15 02:54:36,217 DEBUG [Listener at localhost.localdomain/33693] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@30cdd851, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-15 02:54:36,217 DEBUG [Listener at localhost.localdomain/33693] zookeeper.ReadOnlyZKClient(139): Connect 0x7117e4df to 127.0.0.1:51762 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-15 02:54:36,221 DEBUG [Listener at localhost.localdomain/33693] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@136e067d, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-15 02:54:36,221 DEBUG [Listener at localhost.localdomain/33693] zookeeper.ReadOnlyZKClient(139): Connect 0x5c466b6c to 127.0.0.1:51762 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-15 02:54:36,225 DEBUG [Listener at localhost.localdomain/33693] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@12394d27, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-15 02:54:36,225 DEBUG [Listener at localhost.localdomain/33693] zookeeper.ReadOnlyZKClient(139): Connect 0x46d74611 to 127.0.0.1:51762 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-15 02:54:36,230 DEBUG [Listener at localhost.localdomain/33693] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@29a548ac, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-15 02:54:36,231 DEBUG [Listener at localhost.localdomain/33693] zookeeper.ReadOnlyZKClient(139): Connect 0x3f837278 to 127.0.0.1:51762 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-15 02:54:36,235 DEBUG [Listener at localhost.localdomain/33693] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@43184c57, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-15 02:54:36,235 DEBUG [Listener at localhost.localdomain/33693] zookeeper.ReadOnlyZKClient(139): Connect 0x7e3efec6 to 127.0.0.1:51762 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-15 02:54:36,240 DEBUG [Listener at localhost.localdomain/33693] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@7f456fa2, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-15 02:54:36,240 DEBUG [Listener at localhost.localdomain/33693] zookeeper.ReadOnlyZKClient(139): Connect 0x6b7f716c to 127.0.0.1:51762 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-15 02:54:36,245 DEBUG [Listener at localhost.localdomain/33693] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@62be7a0b, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-15 02:54:36,246 DEBUG [Listener at localhost.localdomain/33693] zookeeper.ReadOnlyZKClient(139): Connect 0x6ade356f to 127.0.0.1:51762 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-15 02:54:36,249 DEBUG [Listener at localhost.localdomain/33693] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@7abacd49, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-15 02:54:36,249 DEBUG [Listener at localhost.localdomain/33693] zookeeper.ReadOnlyZKClient(139): Connect 0x4cbc74e7 to 127.0.0.1:51762 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-15 02:54:36,254 DEBUG [Listener at localhost.localdomain/33693] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@57d25552, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-15 02:54:36,254 DEBUG [Listener at localhost.localdomain/33693] zookeeper.ReadOnlyZKClient(139): Connect 0x3c5d1710 to 127.0.0.1:51762 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-15 02:54:36,275 DEBUG [Listener at localhost.localdomain/33693] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@13695730, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-15 02:54:36,281 DEBUG [hconnection-0xbc9007d-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-15 02:54:36,282 DEBUG [hconnection-0x601a6d5c-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-15 02:54:36,282 DEBUG [hconnection-0xce3b4f7-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-15 02:54:36,284 DEBUG [hconnection-0x24ffc4fd-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-15 02:54:36,284 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:41900, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-15 02:54:36,285 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-15 02:54:36,285 DEBUG [hconnection-0x78b57cc1-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-15 02:54:36,285 DEBUG [hconnection-0x464c8dfb-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-15 02:54:36,286 DEBUG [hconnection-0x3700da32-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-15 02:54:36,286 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:41926, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-15 02:54:36,290 DEBUG [hconnection-0x422827d4-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-15 02:54:36,290 DEBUG [hconnection-0x568e5753-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-15 02:54:36,291 DEBUG [hconnection-0x1d58f670-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-15 02:54:36,294 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:41956, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-15 02:54:36,294 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:41886, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-15 02:54:36,294 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:41890, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-15 02:54:36,294 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:41916, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-15 02:54:36,294 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:41918, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-15 02:54:36,294 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:41934, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-15 02:54:36,294 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:41940, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-15 02:54:36,303 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-15 02:54:36,303 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:41970, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-15 02:54:36,303 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-15 02:54:36,304 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-15 02:54:36,304 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-15 02:54:36,304 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-15 02:54:36,304 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-15 02:54:36,305 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:36,305 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:54:36,307 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-15 02:54:36,307 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:36,307 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-15 02:54:36,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-15 02:54:36,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:54:36,308 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:36,308 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-15 02:54:36,308 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:36,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-15 02:54:36,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:54:36,308 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-15 02:54:36,309 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-15 02:54:36,309 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-15 02:54:36,309 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-15 02:54:36,309 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-15 02:54:36,309 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-15 02:54:36,309 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-15 02:54:36,309 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-15 02:54:36,309 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681527274083.f8160290180ba126c904acc32b60eac7. 2023-04-15 02:54:36,309 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681527274083.f8160290180ba126c904acc32b60eac7. started... 2023-04-15 02:54:36,315 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34681] regionserver.HRegion(9158): Flush requested on f8160290180ba126c904acc32b60eac7 2023-04-15 02:54:36,315 INFO [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing f8160290180ba126c904acc32b60eac7 3/3 column families, dataSize=53.67 KB heapSize=160.13 KB 2023-04-15 02:54:36,315 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681527274083.f8160290180ba126c904acc32b60eac7. as already flushing 2023-04-15 02:54:36,316 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK f8160290180ba126c904acc32b60eac7, store=A 2023-04-15 02:54:36,316 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-15 02:54:36,316 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK f8160290180ba126c904acc32b60eac7, store=B 2023-04-15 02:54:36,316 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-15 02:54:36,316 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK f8160290180ba126c904acc32b60eac7, store=C 2023-04-15 02:54:36,316 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-15 02:54:36,344 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1681527276338, max timestamp=1681527276342Num uniques 859; 2023-04-15 02:54:36,344 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-4] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-15 02:54:36,344 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1681527276338, max timestamp=1681527276342Num uniques 859; 2023-04-15 02:54:36,344 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-1] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-15 02:54:36,345 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681527276338, max timestamp=1681527276343Num uniques 859; 2023-04-15 02:54:36,345 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-7] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-15 02:54:36,378 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f8160290180ba126c904acc32b60eac7, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:54:36,378 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34681] ipc.CallRunner(144): callId: 12 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41918 deadline: 1681527336375, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f8160290180ba126c904acc32b60eac7, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:36,378 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f8160290180ba126c904acc32b60eac7, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:54:36,378 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34681] ipc.CallRunner(144): callId: 13 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41926 deadline: 1681527336375, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f8160290180ba126c904acc32b60eac7, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:36,378 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f8160290180ba126c904acc32b60eac7, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:54:36,379 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34681] ipc.CallRunner(144): callId: 8 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41916 deadline: 1681527336375, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f8160290180ba126c904acc32b60eac7, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:36,379 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f8160290180ba126c904acc32b60eac7, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:54:36,379 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34681] ipc.CallRunner(144): callId: 9 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41890 deadline: 1681527336376, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f8160290180ba126c904acc32b60eac7, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:36,379 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f8160290180ba126c904acc32b60eac7, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:54:36,379 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34681] ipc.CallRunner(144): callId: 8 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41940 deadline: 1681527336376, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f8160290180ba126c904acc32b60eac7, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:36,480 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f8160290180ba126c904acc32b60eac7, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:54:36,481 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34681] ipc.CallRunner(144): callId: 14 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41918 deadline: 1681527336479, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f8160290180ba126c904acc32b60eac7, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:36,481 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f8160290180ba126c904acc32b60eac7, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:54:36,481 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34681] ipc.CallRunner(144): callId: 15 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41926 deadline: 1681527336480, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f8160290180ba126c904acc32b60eac7, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:36,481 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f8160290180ba126c904acc32b60eac7, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:54:36,481 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34681] ipc.CallRunner(144): callId: 10 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41916 deadline: 1681527336480, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f8160290180ba126c904acc32b60eac7, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:36,481 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f8160290180ba126c904acc32b60eac7, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:54:36,481 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34681] ipc.CallRunner(144): callId: 11 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41890 deadline: 1681527336480, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f8160290180ba126c904acc32b60eac7, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:36,481 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f8160290180ba126c904acc32b60eac7, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:54:36,481 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34681] ipc.CallRunner(144): callId: 10 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41940 deadline: 1681527336480, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f8160290180ba126c904acc32b60eac7, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:36,684 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f8160290180ba126c904acc32b60eac7, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:54:36,684 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f8160290180ba126c904acc32b60eac7, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:54:36,684 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34681] ipc.CallRunner(144): callId: 16 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41918 deadline: 1681527336682, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f8160290180ba126c904acc32b60eac7, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:36,684 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34681] ipc.CallRunner(144): callId: 17 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41926 deadline: 1681527336682, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f8160290180ba126c904acc32b60eac7, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:36,684 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f8160290180ba126c904acc32b60eac7, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:54:36,684 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f8160290180ba126c904acc32b60eac7, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:54:36,684 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34681] ipc.CallRunner(144): callId: 12 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41916 deadline: 1681527336682, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f8160290180ba126c904acc32b60eac7, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:36,684 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34681] ipc.CallRunner(144): callId: 13 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41890 deadline: 1681527336682, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f8160290180ba126c904acc32b60eac7, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:36,685 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f8160290180ba126c904acc32b60eac7, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:54:36,685 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34681] ipc.CallRunner(144): callId: 12 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41940 deadline: 1681527336683, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f8160290180ba126c904acc32b60eac7, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:36,759 INFO [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=12 (bloomFilter=true), to=hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/f8160290180ba126c904acc32b60eac7/.tmp/A/40f07466e76f415ab019342c8727995e 2023-04-15 02:54:36,769 INFO [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=12 (bloomFilter=true), to=hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/f8160290180ba126c904acc32b60eac7/.tmp/B/b8c69868004c408eb9d6fd873d117041 2023-04-15 02:54:36,782 INFO [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=12 (bloomFilter=true), to=hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/f8160290180ba126c904acc32b60eac7/.tmp/C/925c7d2aa44e4aa1b8865d4572d399ed 2023-04-15 02:54:36,786 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/f8160290180ba126c904acc32b60eac7/.tmp/A/40f07466e76f415ab019342c8727995e as hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/f8160290180ba126c904acc32b60eac7/A/40f07466e76f415ab019342c8727995e 2023-04-15 02:54:36,791 INFO [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/f8160290180ba126c904acc32b60eac7/A/40f07466e76f415ab019342c8727995e, entries=150, sequenceid=12, filesize=11.6 K 2023-04-15 02:54:36,792 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/f8160290180ba126c904acc32b60eac7/.tmp/B/b8c69868004c408eb9d6fd873d117041 as hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/f8160290180ba126c904acc32b60eac7/B/b8c69868004c408eb9d6fd873d117041 2023-04-15 02:54:36,796 INFO [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/f8160290180ba126c904acc32b60eac7/B/b8c69868004c408eb9d6fd873d117041, entries=150, sequenceid=12, filesize=11.6 K 2023-04-15 02:54:36,797 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/f8160290180ba126c904acc32b60eac7/.tmp/C/925c7d2aa44e4aa1b8865d4572d399ed as hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/f8160290180ba126c904acc32b60eac7/C/925c7d2aa44e4aa1b8865d4572d399ed 2023-04-15 02:54:36,801 INFO [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/f8160290180ba126c904acc32b60eac7/C/925c7d2aa44e4aa1b8865d4572d399ed, entries=150, sequenceid=12, filesize=11.6 K 2023-04-15 02:54:36,802 INFO [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=125.23 KB/128240 for f8160290180ba126c904acc32b60eac7 in 487ms, sequenceid=12, compaction requested=false 2023-04-15 02:54:36,802 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for f8160290180ba126c904acc32b60eac7: 2023-04-15 02:54:36,802 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681527274083.f8160290180ba126c904acc32b60eac7. 2023-04-15 02:54:36,802 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-15 02:54:36,802 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-15 02:54:36,802 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:36,802 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-15 02:54:36,802 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34681,1681527209133' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-15 02:54:36,803 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:36,803 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:36,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:36,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:54:36,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:54:36,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:54:36,804 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:36,804 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-15 02:54:36,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:54:36,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:54:36,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:36,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:54:36,804 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34681,1681527209133' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-15 02:54:36,804 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@64210b96[Count = 0] remaining members to acquire global barrier 2023-04-15 02:54:36,804 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-15 02:54:36,804 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:36,805 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:36,805 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:36,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:36,805 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-15 02:54:36,805 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-15 02:54:36,805 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34681,1681527209133' in zk 2023-04-15 02:54:36,805 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:36,806 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-15 02:54:36,806 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:36,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:36,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:54:36,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:54:36,806 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-15 02:54:36,806 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-15 02:54:36,807 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-15 02:54:36,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:54:36,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:54:36,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:54:36,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:36,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:54:36,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:54:36,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:36,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34681,1681527209133': 2023-04-15 02:54:36,808 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34681,1681527209133' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-15 02:54:36,808 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-15 02:54:36,808 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-15 02:54:36,808 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-15 02:54:36,808 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-15 02:54:36,809 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-15 02:54:36,810 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:36,810 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:36,810 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-15 02:54:36,810 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:36,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:36,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:54:36,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:54:36,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:36,810 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:36,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:54:36,810 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-15 02:54:36,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-15 02:54:36,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:54:36,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:36,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:54:36,811 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:36,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:54:36,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:36,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:54:36,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:54:36,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:36,812 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:36,812 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:54:36,813 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:54:36,813 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:36,813 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-15 02:54:36,813 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:36,813 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-15 02:54:36,813 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-15 02:54:36,813 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-15 02:54:36,813 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:36,813 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-15 02:54:36,813 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-15 02:54:36,813 DEBUG [Thread-1527] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-15 02:54:36,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-15 02:54:36,813 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:36,813 DEBUG [Thread-1527] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-15 02:54:36,813 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-15 02:54:36,813 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-15 02:54:36,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-15 02:54:36,913 DEBUG [Thread-1527] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-15 02:54:36,914 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-15 02:54:36,915 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-15 02:54:36,916 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-15 02:54:36,916 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-15 02:54:36,916 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-15 02:54:36,916 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-15 02:54:36,916 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-15 02:54:36,916 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-15 02:54:36,916 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:36,916 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:54:36,917 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:36,917 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-15 02:54:36,917 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-15 02:54:36,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-15 02:54:36,917 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:36,917 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-15 02:54:36,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:54:36,918 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:36,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-15 02:54:36,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:54:36,918 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-15 02:54:36,918 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-15 02:54:36,918 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-15 02:54:36,918 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-15 02:54:36,918 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-15 02:54:36,918 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-15 02:54:36,918 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-15 02:54:36,918 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-15 02:54:36,918 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681527274083.f8160290180ba126c904acc32b60eac7. 2023-04-15 02:54:36,918 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681527274083.f8160290180ba126c904acc32b60eac7. started... 2023-04-15 02:54:36,919 INFO [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing f8160290180ba126c904acc32b60eac7 3/3 column families, dataSize=125.23 KB heapSize=355.30 KB 2023-04-15 02:54:36,919 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK f8160290180ba126c904acc32b60eac7, store=A 2023-04-15 02:54:36,919 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-15 02:54:36,919 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK f8160290180ba126c904acc32b60eac7, store=B 2023-04-15 02:54:36,919 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-15 02:54:36,919 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK f8160290180ba126c904acc32b60eac7, store=C 2023-04-15 02:54:36,919 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-15 02:54:36,986 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34681] regionserver.HRegion(9158): Flush requested on f8160290180ba126c904acc32b60eac7 2023-04-15 02:54:36,987 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681527274083.f8160290180ba126c904acc32b60eac7. as already flushing 2023-04-15 02:54:37,019 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f8160290180ba126c904acc32b60eac7, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:54:37,019 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34681] ipc.CallRunner(144): callId: 16 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41890 deadline: 1681527337016, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f8160290180ba126c904acc32b60eac7, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:37,020 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f8160290180ba126c904acc32b60eac7, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:54:37,020 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34681] ipc.CallRunner(144): callId: 15 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41940 deadline: 1681527337016, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f8160290180ba126c904acc32b60eac7, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:37,022 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f8160290180ba126c904acc32b60eac7, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:54:37,022 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34681] ipc.CallRunner(144): callId: 16 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41916 deadline: 1681527337019, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f8160290180ba126c904acc32b60eac7, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:37,022 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f8160290180ba126c904acc32b60eac7, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:54:37,022 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34681] ipc.CallRunner(144): callId: 22 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41926 deadline: 1681527337019, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f8160290180ba126c904acc32b60eac7, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:37,022 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f8160290180ba126c904acc32b60eac7, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:54:37,023 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34681] ipc.CallRunner(144): callId: 20 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41918 deadline: 1681527337020, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f8160290180ba126c904acc32b60eac7, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:37,122 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f8160290180ba126c904acc32b60eac7, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:54:37,122 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34681] ipc.CallRunner(144): callId: 18 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41890 deadline: 1681527337120, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f8160290180ba126c904acc32b60eac7, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:37,122 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f8160290180ba126c904acc32b60eac7, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:54:37,122 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34681] ipc.CallRunner(144): callId: 17 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41940 deadline: 1681527337120, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f8160290180ba126c904acc32b60eac7, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:37,125 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f8160290180ba126c904acc32b60eac7, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:54:37,125 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34681] ipc.CallRunner(144): callId: 18 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41916 deadline: 1681527337123, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f8160290180ba126c904acc32b60eac7, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:37,125 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f8160290180ba126c904acc32b60eac7, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:54:37,126 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34681] ipc.CallRunner(144): callId: 24 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41926 deadline: 1681527337124, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f8160290180ba126c904acc32b60eac7, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:37,126 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f8160290180ba126c904acc32b60eac7, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:54:37,126 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34681] ipc.CallRunner(144): callId: 22 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41918 deadline: 1681527337124, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f8160290180ba126c904acc32b60eac7, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:37,323 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f8160290180ba126c904acc32b60eac7, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:54:37,324 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34681] ipc.CallRunner(144): callId: 20 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41890 deadline: 1681527337323, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f8160290180ba126c904acc32b60eac7, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:37,324 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f8160290180ba126c904acc32b60eac7, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:54:37,324 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34681] ipc.CallRunner(144): callId: 19 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41940 deadline: 1681527337323, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f8160290180ba126c904acc32b60eac7, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:37,326 INFO [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=47 (bloomFilter=true), to=hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/f8160290180ba126c904acc32b60eac7/.tmp/A/9d601bd6d01f4a3f808814012e1a1b12 2023-04-15 02:54:37,328 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f8160290180ba126c904acc32b60eac7, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:54:37,328 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34681] ipc.CallRunner(144): callId: 26 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41926 deadline: 1681527337326, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f8160290180ba126c904acc32b60eac7, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:37,328 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f8160290180ba126c904acc32b60eac7, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:54:37,328 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34681] ipc.CallRunner(144): callId: 20 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41916 deadline: 1681527337326, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f8160290180ba126c904acc32b60eac7, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:37,331 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f8160290180ba126c904acc32b60eac7, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:54:37,331 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34681] ipc.CallRunner(144): callId: 24 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41918 deadline: 1681527337330, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f8160290180ba126c904acc32b60eac7, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:37,627 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f8160290180ba126c904acc32b60eac7, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:54:37,627 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34681] ipc.CallRunner(144): callId: 21 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41940 deadline: 1681527337625, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f8160290180ba126c904acc32b60eac7, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:37,627 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f8160290180ba126c904acc32b60eac7, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:54:37,627 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34681] ipc.CallRunner(144): callId: 22 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41890 deadline: 1681527337625, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f8160290180ba126c904acc32b60eac7, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:37,630 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f8160290180ba126c904acc32b60eac7, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:54:37,630 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34681] ipc.CallRunner(144): callId: 28 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41926 deadline: 1681527337629, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f8160290180ba126c904acc32b60eac7, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:37,630 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f8160290180ba126c904acc32b60eac7, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:54:37,631 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34681] ipc.CallRunner(144): callId: 22 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41916 deadline: 1681527337629, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f8160290180ba126c904acc32b60eac7, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:37,634 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f8160290180ba126c904acc32b60eac7, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:54:37,634 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34681] ipc.CallRunner(144): callId: 26 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41918 deadline: 1681527337633, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f8160290180ba126c904acc32b60eac7, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:37,736 INFO [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=47 (bloomFilter=true), to=hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/f8160290180ba126c904acc32b60eac7/.tmp/B/abc1edeb35f04e01b9b1176e384165f7 2023-04-15 02:54:38,128 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f8160290180ba126c904acc32b60eac7, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:54:38,128 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34681] ipc.CallRunner(144): callId: 23 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41940 deadline: 1681527338128, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f8160290180ba126c904acc32b60eac7, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:38,131 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f8160290180ba126c904acc32b60eac7, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:54:38,131 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34681] ipc.CallRunner(144): callId: 24 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41890 deadline: 1681527338130, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f8160290180ba126c904acc32b60eac7, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:38,135 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f8160290180ba126c904acc32b60eac7, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:54:38,135 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34681] ipc.CallRunner(144): callId: 24 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41916 deadline: 1681527338133, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f8160290180ba126c904acc32b60eac7, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:38,137 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f8160290180ba126c904acc32b60eac7, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:54:38,137 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f8160290180ba126c904acc32b60eac7, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:54:38,137 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34681] ipc.CallRunner(144): callId: 30 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41926 deadline: 1681527338135, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f8160290180ba126c904acc32b60eac7, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:38,137 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34681] ipc.CallRunner(144): callId: 28 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41918 deadline: 1681527338135, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f8160290180ba126c904acc32b60eac7, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:38,147 INFO [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=40.25 KB at sequenceid=47 (bloomFilter=true), to=hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/f8160290180ba126c904acc32b60eac7/.tmp/C/1c397bc1e228408e8cd10a50d92af44f 2023-04-15 02:54:38,151 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/f8160290180ba126c904acc32b60eac7/.tmp/A/9d601bd6d01f4a3f808814012e1a1b12 as hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/f8160290180ba126c904acc32b60eac7/A/9d601bd6d01f4a3f808814012e1a1b12 2023-04-15 02:54:38,154 INFO [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/f8160290180ba126c904acc32b60eac7/A/9d601bd6d01f4a3f808814012e1a1b12, entries=150, sequenceid=47, filesize=11.6 K 2023-04-15 02:54:38,155 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/f8160290180ba126c904acc32b60eac7/.tmp/B/abc1edeb35f04e01b9b1176e384165f7 as hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/f8160290180ba126c904acc32b60eac7/B/abc1edeb35f04e01b9b1176e384165f7 2023-04-15 02:54:38,158 INFO [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/f8160290180ba126c904acc32b60eac7/B/abc1edeb35f04e01b9b1176e384165f7, entries=150, sequenceid=47, filesize=11.6 K 2023-04-15 02:54:38,159 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/f8160290180ba126c904acc32b60eac7/.tmp/C/1c397bc1e228408e8cd10a50d92af44f as hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/f8160290180ba126c904acc32b60eac7/C/1c397bc1e228408e8cd10a50d92af44f 2023-04-15 02:54:38,162 INFO [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/f8160290180ba126c904acc32b60eac7/C/1c397bc1e228408e8cd10a50d92af44f, entries=150, sequenceid=47, filesize=11.6 K 2023-04-15 02:54:38,163 INFO [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~125.23 KB/128240, heapSize ~355.25 KB/363776, currentSize=60.38 KB/61830 for f8160290180ba126c904acc32b60eac7 in 1245ms, sequenceid=47, compaction requested=false 2023-04-15 02:54:38,163 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for f8160290180ba126c904acc32b60eac7: 2023-04-15 02:54:38,163 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681527274083.f8160290180ba126c904acc32b60eac7. 2023-04-15 02:54:38,163 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-15 02:54:38,163 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-15 02:54:38,163 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:38,163 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-15 02:54:38,164 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34681,1681527209133' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-15 02:54:38,164 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:38,164 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:38,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:38,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:54:38,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:54:38,165 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:38,165 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-15 02:54:38,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:54:38,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:54:38,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:54:38,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:38,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:54:38,166 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34681,1681527209133' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-15 02:54:38,166 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@32a443a0[Count = 0] remaining members to acquire global barrier 2023-04-15 02:54:38,166 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-15 02:54:38,166 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:38,166 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:38,166 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:38,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:38,167 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-15 02:54:38,167 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-15 02:54:38,167 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34681,1681527209133' in zk 2023-04-15 02:54:38,167 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:38,167 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-15 02:54:38,167 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-15 02:54:38,167 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-15 02:54:38,167 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:38,167 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-15 02:54:38,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:38,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:54:38,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:54:38,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:54:38,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:54:38,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:54:38,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:38,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:54:38,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:54:38,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:38,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34681,1681527209133': 2023-04-15 02:54:38,169 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34681,1681527209133' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-15 02:54:38,169 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-15 02:54:38,169 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-15 02:54:38,169 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-15 02:54:38,169 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-15 02:54:38,169 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-15 02:54:38,170 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:38,170 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:38,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:38,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:54:38,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:54:38,170 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-15 02:54:38,170 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:38,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:38,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:54:38,170 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-15 02:54:38,170 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:38,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-15 02:54:38,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:54:38,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:38,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:54:38,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:54:38,171 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:38,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:38,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:54:38,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:54:38,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:38,173 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:38,173 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:54:38,173 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:54:38,173 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:38,173 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-15 02:54:38,173 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-15 02:54:38,173 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-15 02:54:38,173 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:38,173 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-15 02:54:38,173 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:38,173 DEBUG [Thread-1527] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-15 02:54:38,173 DEBUG [Thread-1527] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-15 02:54:38,173 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-15 02:54:38,173 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-15 02:54:38,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-15 02:54:38,173 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:38,173 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-15 02:54:38,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-15 02:54:38,273 DEBUG [Thread-1527] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-15 02:54:38,274 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-15 02:54:38,275 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-15 02:54:38,276 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-15 02:54:38,276 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-15 02:54:38,276 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-15 02:54:38,276 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-15 02:54:38,276 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-15 02:54:38,276 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-15 02:54:38,276 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:38,276 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:54:38,277 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:38,277 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-15 02:54:38,277 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-15 02:54:38,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-15 02:54:38,277 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:38,277 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-15 02:54:38,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:54:38,277 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:38,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-15 02:54:38,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:54:38,278 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-15 02:54:38,278 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-15 02:54:38,278 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-15 02:54:38,278 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-15 02:54:38,278 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-15 02:54:38,278 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-15 02:54:38,278 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-15 02:54:38,278 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681527274083.f8160290180ba126c904acc32b60eac7. 2023-04-15 02:54:38,278 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681527274083.f8160290180ba126c904acc32b60eac7. started... 2023-04-15 02:54:38,278 INFO [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing f8160290180ba126c904acc32b60eac7 3/3 column families, dataSize=60.38 KB heapSize=180.05 KB 2023-04-15 02:54:38,279 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK f8160290180ba126c904acc32b60eac7, store=A 2023-04-15 02:54:38,279 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-15 02:54:38,279 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK f8160290180ba126c904acc32b60eac7, store=B 2023-04-15 02:54:38,279 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-15 02:54:38,279 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK f8160290180ba126c904acc32b60eac7, store=C 2023-04-15 02:54:38,279 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-15 02:54:38,285 INFO [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=59 (bloomFilter=true), to=hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/f8160290180ba126c904acc32b60eac7/.tmp/A/f4e10dcc69934f22941dd7fde1d40ee1 2023-04-15 02:54:38,698 INFO [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=59 (bloomFilter=true), to=hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/f8160290180ba126c904acc32b60eac7/.tmp/B/e8ba002fe0f945c0b9561495f7b6e3a5 2023-04-15 02:54:38,707 INFO [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=59 (bloomFilter=true), to=hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/f8160290180ba126c904acc32b60eac7/.tmp/C/f4d03398fbf44a50944d5ba548a577d2 2023-04-15 02:54:38,722 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/f8160290180ba126c904acc32b60eac7/.tmp/A/f4e10dcc69934f22941dd7fde1d40ee1 as hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/f8160290180ba126c904acc32b60eac7/A/f4e10dcc69934f22941dd7fde1d40ee1 2023-04-15 02:54:38,726 INFO [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/f8160290180ba126c904acc32b60eac7/A/f4e10dcc69934f22941dd7fde1d40ee1, entries=150, sequenceid=59, filesize=11.6 K 2023-04-15 02:54:38,726 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/f8160290180ba126c904acc32b60eac7/.tmp/B/e8ba002fe0f945c0b9561495f7b6e3a5 as hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/f8160290180ba126c904acc32b60eac7/B/e8ba002fe0f945c0b9561495f7b6e3a5 2023-04-15 02:54:38,732 INFO [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/f8160290180ba126c904acc32b60eac7/B/e8ba002fe0f945c0b9561495f7b6e3a5, entries=150, sequenceid=59, filesize=11.6 K 2023-04-15 02:54:38,732 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/f8160290180ba126c904acc32b60eac7/.tmp/C/f4d03398fbf44a50944d5ba548a577d2 as hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/f8160290180ba126c904acc32b60eac7/C/f4d03398fbf44a50944d5ba548a577d2 2023-04-15 02:54:38,737 INFO [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/f8160290180ba126c904acc32b60eac7/C/f4d03398fbf44a50944d5ba548a577d2, entries=150, sequenceid=59, filesize=11.6 K 2023-04-15 02:54:38,738 INFO [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~60.38 KB/61830, heapSize ~180 KB/184320, currentSize=0 B/0 for f8160290180ba126c904acc32b60eac7 in 460ms, sequenceid=59, compaction requested=true 2023-04-15 02:54:38,738 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for f8160290180ba126c904acc32b60eac7: 2023-04-15 02:54:38,738 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681527274083.f8160290180ba126c904acc32b60eac7. 2023-04-15 02:54:38,738 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-15 02:54:38,738 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-15 02:54:38,738 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:38,738 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-15 02:54:38,738 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34681,1681527209133' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-15 02:54:38,739 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:38,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:38,739 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:38,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:54:38,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:54:38,739 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:38,740 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-15 02:54:38,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:54:38,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:54:38,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:54:38,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:38,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:54:38,741 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34681,1681527209133' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-15 02:54:38,741 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4a98c378[Count = 0] remaining members to acquire global barrier 2023-04-15 02:54:38,741 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-15 02:54:38,741 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:38,741 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:38,741 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:38,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:38,741 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-15 02:54:38,741 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-15 02:54:38,741 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34681,1681527209133' in zk 2023-04-15 02:54:38,741 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:38,742 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-15 02:54:38,742 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:38,742 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-15 02:54:38,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:38,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:54:38,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:54:38,742 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-15 02:54:38,743 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-15 02:54:38,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:54:38,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:54:38,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:54:38,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:38,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:54:38,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:54:38,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:38,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34681,1681527209133': 2023-04-15 02:54:38,745 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34681,1681527209133' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-15 02:54:38,745 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-15 02:54:38,745 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-15 02:54:38,745 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-15 02:54:38,745 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-15 02:54:38,745 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-15 02:54:38,746 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:38,746 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:38,746 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:38,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:38,746 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-15 02:54:38,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:38,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:54:38,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:54:38,753 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-15 02:54:38,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:54:38,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-15 02:54:38,753 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:38,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:54:38,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:38,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:54:38,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:54:38,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:38,755 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:38,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:54:38,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:54:38,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:38,762 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:38,762 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:54:38,762 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-15 02:54:38,762 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-15 02:54:38,762 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-15 02:54:38,762 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:54:38,762 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-15 02:54:38,762 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-15 02:54:38,762 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-15 02:54:38,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-15 02:54:38,762 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-15 02:54:38,762 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:38,762 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:38,762 DEBUG [Thread-1527] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-15 02:54:38,762 DEBUG [Thread-1527] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-15 02:54:38,762 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:38,762 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:38,762 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-15 02:54:38,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-15 02:54:38,862 DEBUG [Thread-1527] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-15 02:54:38,863 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-15 02:54:38,865 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-15 02:54:38,870 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-15 02:54:38,870 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-15 02:54:38,870 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-15 02:54:38,870 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-15 02:54:38,870 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-15 02:54:38,870 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-15 02:54:38,870 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:38,871 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:54:38,871 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-15 02:54:38,871 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-15 02:54:38,871 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:38,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-15 02:54:38,872 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:38,872 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-15 02:54:38,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:54:38,872 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:38,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-15 02:54:38,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:54:38,872 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-15 02:54:38,872 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-15 02:54:38,872 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-15 02:54:38,873 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-15 02:54:38,873 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-15 02:54:38,873 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-15 02:54:38,874 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-15 02:54:38,874 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-15 02:54:38,874 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681527274083.f8160290180ba126c904acc32b60eac7. 2023-04-15 02:54:38,874 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681527274083.f8160290180ba126c904acc32b60eac7. started... 2023-04-15 02:54:38,874 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for f8160290180ba126c904acc32b60eac7: 2023-04-15 02:54:38,874 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681527274083.f8160290180ba126c904acc32b60eac7. 2023-04-15 02:54:38,874 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-15 02:54:38,874 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-15 02:54:38,874 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:38,874 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-15 02:54:38,874 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34681,1681527209133' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-15 02:54:38,876 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:38,876 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:38,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:38,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:54:38,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:54:38,876 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:38,877 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-15 02:54:38,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:54:38,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:54:38,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:54:38,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:38,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:54:38,878 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34681,1681527209133' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-15 02:54:38,878 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3c6fabc5[Count = 0] remaining members to acquire global barrier 2023-04-15 02:54:38,878 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-15 02:54:38,878 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:38,878 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:38,878 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:38,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:38,878 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:38,878 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-15 02:54:38,878 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-15 02:54:38,879 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-15 02:54:38,879 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34681,1681527209133' in zk 2023-04-15 02:54:38,879 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:38,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:38,879 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-15 02:54:38,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:54:38,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:54:38,879 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-15 02:54:38,880 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-15 02:54:38,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:54:38,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:54:38,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:54:38,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:38,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:54:38,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:54:38,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:38,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34681,1681527209133': 2023-04-15 02:54:38,881 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34681,1681527209133' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-15 02:54:38,881 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-15 02:54:38,881 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-15 02:54:38,881 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-15 02:54:38,881 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-15 02:54:38,881 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-15 02:54:38,882 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:38,882 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-15 02:54:38,882 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:38,882 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:38,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:38,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:38,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:54:38,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:54:38,883 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-15 02:54:38,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-15 02:54:38,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:54:38,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:38,883 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:38,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:54:38,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:54:38,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:54:38,884 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:38,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:38,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:54:38,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:54:38,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:38,886 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-15 02:54:38,886 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:38,886 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-15 02:54:38,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-15 02:54:38,886 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-15 02:54:38,886 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-15 02:54:38,886 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-15 02:54:38,886 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:54:38,886 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-15 02:54:38,886 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-15 02:54:38,886 DEBUG [Thread-1527] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-15 02:54:38,886 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:54:38,886 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-15 02:54:38,886 DEBUG [Thread-1527] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-15 02:54:38,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-15 02:54:38,886 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:38,886 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:38,887 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:38,887 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:38,986 DEBUG [Thread-1527] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-15 02:54:38,987 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-15 02:54:38,988 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-15 02:54:38,989 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-15 02:54:38,990 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-15 02:54:38,990 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-15 02:54:38,990 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-15 02:54:38,990 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-15 02:54:38,990 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-15 02:54:38,990 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:38,990 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:54:38,991 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:38,991 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-15 02:54:38,991 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-15 02:54:38,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-15 02:54:38,991 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:38,991 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-15 02:54:38,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:54:38,991 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:38,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-15 02:54:38,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:54:38,992 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-15 02:54:38,992 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-15 02:54:38,992 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-15 02:54:38,992 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-15 02:54:38,992 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-15 02:54:38,992 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-15 02:54:38,992 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-15 02:54:38,992 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-15 02:54:38,992 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681527274083.f8160290180ba126c904acc32b60eac7. 2023-04-15 02:54:38,992 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681527274083.f8160290180ba126c904acc32b60eac7. started... 2023-04-15 02:54:38,993 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for f8160290180ba126c904acc32b60eac7: 2023-04-15 02:54:38,993 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681527274083.f8160290180ba126c904acc32b60eac7. 2023-04-15 02:54:38,993 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-15 02:54:38,993 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-15 02:54:38,993 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:38,993 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-15 02:54:38,993 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34681,1681527209133' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-15 02:54:38,994 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:38,994 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:38,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:38,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:54:38,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:54:38,994 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:38,994 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-15 02:54:38,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:54:38,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:54:38,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:54:38,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:38,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:54:38,995 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34681,1681527209133' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-15 02:54:38,995 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-15 02:54:38,995 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@507e7a65[Count = 0] remaining members to acquire global barrier 2023-04-15 02:54:38,995 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:38,995 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:38,996 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:38,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:38,996 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-15 02:54:38,996 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-15 02:54:38,996 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34681,1681527209133' in zk 2023-04-15 02:54:38,996 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:38,996 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-15 02:54:38,996 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-15 02:54:38,996 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-15 02:54:38,996 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:38,996 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-15 02:54:38,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:38,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:54:38,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:54:38,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:54:38,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:54:38,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:54:38,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:38,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:54:38,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:54:38,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:38,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34681,1681527209133': 2023-04-15 02:54:38,999 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34681,1681527209133' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-15 02:54:38,999 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-15 02:54:38,999 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-15 02:54:38,999 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-15 02:54:38,999 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-15 02:54:38,999 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-15 02:54:39,000 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:39,000 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-15 02:54:39,000 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:39,000 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:39,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:39,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:39,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:54:39,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:54:39,000 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-15 02:54:39,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-15 02:54:39,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:54:39,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:39,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:54:39,001 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:39,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:54:39,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:54:39,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:39,001 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:39,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:54:39,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:54:39,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:39,003 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:39,003 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-15 02:54:39,003 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:54:39,003 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-15 02:54:39,003 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-15 02:54:39,003 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-15 02:54:39,003 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:54:39,003 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-15 02:54:39,003 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-15 02:54:39,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-15 02:54:39,003 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:39,003 DEBUG [Thread-1527] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-15 02:54:39,004 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:39,004 DEBUG [Thread-1527] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-15 02:54:39,004 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:39,004 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:39,004 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-15 02:54:39,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-15 02:54:39,104 DEBUG [Thread-1527] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-15 02:54:39,104 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-15 02:54:39,109 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-15 02:54:39,113 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-15 02:54:39,113 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-15 02:54:39,113 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-15 02:54:39,113 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-15 02:54:39,113 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-15 02:54:39,113 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-15 02:54:39,114 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:39,114 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:54:39,114 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:39,114 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-15 02:54:39,114 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-15 02:54:39,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-15 02:54:39,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:54:39,115 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:39,115 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-15 02:54:39,115 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:39,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-15 02:54:39,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:54:39,115 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-15 02:54:39,116 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-15 02:54:39,116 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-15 02:54:39,116 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-15 02:54:39,116 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-15 02:54:39,116 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-15 02:54:39,117 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-15 02:54:39,117 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-15 02:54:39,117 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681527274083.f8160290180ba126c904acc32b60eac7. 2023-04-15 02:54:39,117 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681527274083.f8160290180ba126c904acc32b60eac7. started... 2023-04-15 02:54:39,117 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for f8160290180ba126c904acc32b60eac7: 2023-04-15 02:54:39,117 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681527274083.f8160290180ba126c904acc32b60eac7. 2023-04-15 02:54:39,117 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-15 02:54:39,118 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-15 02:54:39,118 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:39,118 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-15 02:54:39,118 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34681,1681527209133' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-15 02:54:39,119 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:39,119 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:39,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:39,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:54:39,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:54:39,119 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:39,119 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-15 02:54:39,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:54:39,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:54:39,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:54:39,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:39,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:54:39,128 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34681,1681527209133' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-15 02:54:39,128 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@40d79a89[Count = 0] remaining members to acquire global barrier 2023-04-15 02:54:39,128 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-15 02:54:39,128 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:39,137 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:39,137 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:39,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:39,137 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-15 02:54:39,137 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-15 02:54:39,137 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34681,1681527209133' in zk 2023-04-15 02:54:39,141 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:39,141 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-15 02:54:39,141 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-15 02:54:39,141 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-15 02:54:39,141 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34681,1681527209133' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-15 02:54:39,141 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-15 02:54:39,141 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-15 02:54:39,141 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-15 02:54:39,141 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-15 02:54:39,141 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-15 02:54:39,141 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-15 02:54:39,142 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:39,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:39,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:54:39,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:54:39,143 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:39,143 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-15 02:54:39,143 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:39,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:39,143 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-15 02:54:39,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-15 02:54:39,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:39,144 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34681] regionserver.HRegion(9158): Flush requested on f8160290180ba126c904acc32b60eac7 2023-04-15 02:54:39,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:54:39,144 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing f8160290180ba126c904acc32b60eac7 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-15 02:54:39,144 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:39,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:54:39,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:54:39,145 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK f8160290180ba126c904acc32b60eac7, store=A 2023-04-15 02:54:39,145 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-15 02:54:39,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:54:39,145 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK f8160290180ba126c904acc32b60eac7, store=B 2023-04-15 02:54:39,145 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-15 02:54:39,145 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK f8160290180ba126c904acc32b60eac7, store=C 2023-04-15 02:54:39,145 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-15 02:54:39,145 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:39,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:39,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:54:39,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:54:39,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:39,147 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:39,147 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:54:39,147 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:54:39,147 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:39,148 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:39,148 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:39,148 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-15 02:54:39,148 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-15 02:54:39,148 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-15 02:54:39,148 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-15 02:54:39,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-15 02:54:39,148 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:39,148 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-15 02:54:39,148 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-15 02:54:39,148 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-15 02:54:39,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-15 02:54:39,149 DEBUG [Thread-1527] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-15 02:54:39,149 DEBUG [Thread-1527] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-15 02:54:39,169 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f8160290180ba126c904acc32b60eac7, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:54:39,170 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34681] ipc.CallRunner(144): callId: 32 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41940 deadline: 1681527339163, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f8160290180ba126c904acc32b60eac7, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:39,170 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f8160290180ba126c904acc32b60eac7, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:54:39,170 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34681] ipc.CallRunner(144): callId: 31 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41916 deadline: 1681527339164, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f8160290180ba126c904acc32b60eac7, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:39,170 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1681527279155, max timestamp=1681527279168Num uniques 859; 2023-04-15 02:54:39,170 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1681527279155, max timestamp=1681527279168Num uniques 859; 2023-04-15 02:54:39,170 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-15 02:54:39,170 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-6] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-15 02:54:39,171 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1681527279155, max timestamp=1681527279168Num uniques 859; 2023-04-15 02:54:39,171 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-2] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-04-15 02:54:39,229 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f8160290180ba126c904acc32b60eac7, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:54:39,229 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34681] ipc.CallRunner(144): callId: 41 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41926 deadline: 1681527339225, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f8160290180ba126c904acc32b60eac7, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:39,237 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f8160290180ba126c904acc32b60eac7, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:54:39,238 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34681] ipc.CallRunner(144): callId: 37 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41890 deadline: 1681527339234, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f8160290180ba126c904acc32b60eac7, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:39,238 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f8160290180ba126c904acc32b60eac7, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:54:39,238 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34681] ipc.CallRunner(144): callId: 38 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41918 deadline: 1681527339234, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f8160290180ba126c904acc32b60eac7, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:39,249 DEBUG [Thread-1527] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-15 02:54:39,250 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-15 02:54:39,251 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-15 02:54:39,252 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-15 02:54:39,252 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-15 02:54:39,252 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-15 02:54:39,252 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-15 02:54:39,252 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-15 02:54:39,252 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-15 02:54:39,253 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:39,253 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:54:39,253 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-15 02:54:39,253 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:39,254 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-15 02:54:39,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-15 02:54:39,254 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:39,254 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-15 02:54:39,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:54:39,254 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:39,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-15 02:54:39,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:54:39,254 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-15 02:54:39,258 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-15 02:54:39,258 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-15 02:54:39,258 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-15 02:54:39,258 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-15 02:54:39,258 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-15 02:54:39,258 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-15 02:54:39,258 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-15 02:54:39,258 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681527274083.f8160290180ba126c904acc32b60eac7. 2023-04-15 02:54:39,258 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681527274083.f8160290180ba126c904acc32b60eac7. started... 2023-04-15 02:54:39,258 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681527274083.f8160290180ba126c904acc32b60eac7. as already flushing 2023-04-15 02:54:39,258 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681527274083.f8160290180ba126c904acc32b60eac7. 2023-04-15 02:54:39,259 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-15 02:54:39,259 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-15 02:54:39,259 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:39,259 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-15 02:54:39,259 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34681,1681527209133' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-15 02:54:39,266 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:39,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:39,266 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:39,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:54:39,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:54:39,266 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:39,266 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-15 02:54:39,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:54:39,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:54:39,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:54:39,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:39,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:54:39,268 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34681,1681527209133' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-15 02:54:39,268 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4c5802e8[Count = 0] remaining members to acquire global barrier 2023-04-15 02:54:39,268 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-15 02:54:39,268 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:39,270 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:39,270 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:39,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:39,270 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-15 02:54:39,270 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-15 02:54:39,270 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34681,1681527209133' in zk 2023-04-15 02:54:39,270 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:39,270 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-15 02:54:39,272 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:39,272 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-15 02:54:39,272 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-15 02:54:39,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:39,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:54:39,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:54:39,272 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-15 02:54:39,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:54:39,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:54:39,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:54:39,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:39,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:54:39,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:54:39,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:39,274 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f8160290180ba126c904acc32b60eac7, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:54:39,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34681,1681527209133': 2023-04-15 02:54:39,274 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34681] ipc.CallRunner(144): callId: 34 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41940 deadline: 1681527339271, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f8160290180ba126c904acc32b60eac7, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:39,274 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34681,1681527209133' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-15 02:54:39,274 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-15 02:54:39,274 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-15 02:54:39,274 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-15 02:54:39,274 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-15 02:54:39,274 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f8160290180ba126c904acc32b60eac7, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:54:39,274 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-15 02:54:39,274 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34681] ipc.CallRunner(144): callId: 33 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41916 deadline: 1681527339271, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f8160290180ba126c904acc32b60eac7, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:39,275 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:39,275 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:39,275 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:39,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:39,275 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-15 02:54:39,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:39,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:54:39,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:54:39,276 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-15 02:54:39,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-15 02:54:39,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:54:39,276 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:39,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:39,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:54:39,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:54:39,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:54:39,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:39,277 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:39,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:54:39,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:54:39,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:39,278 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:39,278 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:54:39,278 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:54:39,278 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:39,278 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-15 02:54:39,278 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-15 02:54:39,278 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:39,278 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-15 02:54:39,278 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-15 02:54:39,278 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-15 02:54:39,279 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-15 02:54:39,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-15 02:54:39,279 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-15 02:54:39,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-15 02:54:39,279 DEBUG [Thread-1527] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-15 02:54:39,279 DEBUG [Thread-1527] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-15 02:54:39,278 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:39,279 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:39,333 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f8160290180ba126c904acc32b60eac7, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:54:39,333 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34681] ipc.CallRunner(144): callId: 43 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41926 deadline: 1681527339331, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f8160290180ba126c904acc32b60eac7, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:39,341 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f8160290180ba126c904acc32b60eac7, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:54:39,341 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34681] ipc.CallRunner(144): callId: 39 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41890 deadline: 1681527339339, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f8160290180ba126c904acc32b60eac7, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:39,341 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f8160290180ba126c904acc32b60eac7, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:54:39,341 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34681] ipc.CallRunner(144): callId: 40 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41918 deadline: 1681527339340, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f8160290180ba126c904acc32b60eac7, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:39,365 WARN [HBase-Metrics2-1] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties 2023-04-15 02:54:39,379 DEBUG [Thread-1527] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-15 02:54:39,380 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-15 02:54:39,381 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-15 02:54:39,382 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-15 02:54:39,382 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-15 02:54:39,382 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-15 02:54:39,382 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-15 02:54:39,382 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-15 02:54:39,382 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-15 02:54:39,382 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:39,382 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:54:39,383 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:39,383 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-15 02:54:39,383 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-15 02:54:39,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-15 02:54:39,383 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:39,383 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-15 02:54:39,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:54:39,384 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:39,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-15 02:54:39,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:54:39,384 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-15 02:54:39,385 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-15 02:54:39,385 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-15 02:54:39,385 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-15 02:54:39,385 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-15 02:54:39,385 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-15 02:54:39,394 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-15 02:54:39,394 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681527274083.f8160290180ba126c904acc32b60eac7. 2023-04-15 02:54:39,394 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-15 02:54:39,394 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681527274083.f8160290180ba126c904acc32b60eac7. started... 2023-04-15 02:54:39,394 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681527274083.f8160290180ba126c904acc32b60eac7. as already flushing 2023-04-15 02:54:39,394 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681527274083.f8160290180ba126c904acc32b60eac7. 2023-04-15 02:54:39,394 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-15 02:54:39,394 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-15 02:54:39,394 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:39,394 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-15 02:54:39,394 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34681,1681527209133' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-15 02:54:39,395 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:39,395 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:39,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:39,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:54:39,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:54:39,396 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:39,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:54:39,396 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-15 02:54:39,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:54:39,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:54:39,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:39,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:54:39,397 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34681,1681527209133' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-15 02:54:39,397 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@da8a925[Count = 0] remaining members to acquire global barrier 2023-04-15 02:54:39,397 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-15 02:54:39,397 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:39,398 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:39,398 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:39,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:39,398 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-15 02:54:39,398 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-15 02:54:39,398 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34681,1681527209133' in zk 2023-04-15 02:54:39,398 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:39,398 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-15 02:54:39,399 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-15 02:54:39,399 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-15 02:54:39,399 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:39,399 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-15 02:54:39,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:39,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:54:39,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:54:39,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:54:39,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:54:39,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:54:39,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:39,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:54:39,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:54:39,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:39,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34681,1681527209133': 2023-04-15 02:54:39,400 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34681,1681527209133' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-15 02:54:39,400 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-15 02:54:39,401 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-15 02:54:39,401 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-15 02:54:39,401 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-15 02:54:39,401 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-15 02:54:39,401 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:39,401 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:39,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:39,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:54:39,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:54:39,401 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:39,401 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-15 02:54:39,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:39,402 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:39,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:54:39,402 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-15 02:54:39,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-15 02:54:39,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:54:39,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:39,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:54:39,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:54:39,402 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:39,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:39,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:54:39,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:54:39,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:39,404 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:39,404 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:54:39,404 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:54:39,404 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:39,404 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-15 02:54:39,404 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-15 02:54:39,404 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-15 02:54:39,404 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-15 02:54:39,404 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-15 02:54:39,404 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-15 02:54:39,404 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:39,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-15 02:54:39,404 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:39,404 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-15 02:54:39,404 DEBUG [Thread-1527] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-15 02:54:39,404 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:39,404 DEBUG [Thread-1527] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-15 02:54:39,404 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-15 02:54:39,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-15 02:54:39,477 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f8160290180ba126c904acc32b60eac7, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:54:39,477 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34681] ipc.CallRunner(144): callId: 35 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41916 deadline: 1681527339475, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f8160290180ba126c904acc32b60eac7, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:39,478 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f8160290180ba126c904acc32b60eac7, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:54:39,478 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34681] ipc.CallRunner(144): callId: 36 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41940 deadline: 1681527339476, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f8160290180ba126c904acc32b60eac7, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:39,504 DEBUG [Thread-1527] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-15 02:54:39,505 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-15 02:54:39,506 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-15 02:54:39,507 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-15 02:54:39,507 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-15 02:54:39,507 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-15 02:54:39,507 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-15 02:54:39,507 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-15 02:54:39,507 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-15 02:54:39,508 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:39,508 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:54:39,508 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-15 02:54:39,508 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:39,509 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-15 02:54:39,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-15 02:54:39,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:54:39,509 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:39,509 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-15 02:54:39,510 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:39,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-15 02:54:39,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:54:39,510 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-15 02:54:39,510 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-15 02:54:39,510 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-15 02:54:39,510 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-15 02:54:39,511 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-15 02:54:39,511 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-15 02:54:39,511 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-15 02:54:39,511 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-15 02:54:39,511 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681527274083.f8160290180ba126c904acc32b60eac7. 2023-04-15 02:54:39,511 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681527274083.f8160290180ba126c904acc32b60eac7. started... 2023-04-15 02:54:39,512 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681527274083.f8160290180ba126c904acc32b60eac7. as already flushing 2023-04-15 02:54:39,512 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681527274083.f8160290180ba126c904acc32b60eac7. 2023-04-15 02:54:39,512 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-15 02:54:39,512 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-15 02:54:39,512 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:39,512 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-15 02:54:39,512 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34681,1681527209133' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-15 02:54:39,513 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:39,513 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:39,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:39,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:54:39,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:54:39,514 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:39,514 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-15 02:54:39,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:54:39,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:54:39,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:54:39,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:39,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:54:39,515 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34681,1681527209133' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-15 02:54:39,515 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2e2657a7[Count = 0] remaining members to acquire global barrier 2023-04-15 02:54:39,515 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-15 02:54:39,515 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:39,515 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:39,515 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:39,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:39,515 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-15 02:54:39,516 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-15 02:54:39,516 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34681,1681527209133' in zk 2023-04-15 02:54:39,516 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:39,516 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-15 02:54:39,516 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-15 02:54:39,516 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:39,516 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-15 02:54:39,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:39,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:54:39,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:54:39,517 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-15 02:54:39,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:54:39,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:54:39,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:54:39,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:39,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:54:39,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:54:39,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:39,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34681,1681527209133': 2023-04-15 02:54:39,519 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34681,1681527209133' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-15 02:54:39,519 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-15 02:54:39,519 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-15 02:54:39,519 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-15 02:54:39,519 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-15 02:54:39,519 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-15 02:54:39,520 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:39,520 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:39,520 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-15 02:54:39,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:39,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:54:39,520 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:39,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:39,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:54:39,520 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-15 02:54:39,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-15 02:54:39,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:54:39,520 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:39,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:54:39,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:39,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:54:39,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:54:39,521 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:39,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:39,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:54:39,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:54:39,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:39,525 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-15 02:54:39,525 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-15 02:54:39,525 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-15 02:54:39,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-15 02:54:39,525 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:39,525 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:54:39,525 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:54:39,525 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:39,525 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:39,525 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-15 02:54:39,525 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-15 02:54:39,525 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-15 02:54:39,525 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-15 02:54:39,526 DEBUG [Thread-1527] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-15 02:54:39,526 DEBUG [Thread-1527] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-15 02:54:39,525 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:39,526 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:39,526 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-15 02:54:39,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-15 02:54:39,538 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f8160290180ba126c904acc32b60eac7, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:54:39,538 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34681] ipc.CallRunner(144): callId: 45 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41926 deadline: 1681527339535, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f8160290180ba126c904acc32b60eac7, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:39,543 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f8160290180ba126c904acc32b60eac7, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:54:39,543 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34681] ipc.CallRunner(144): callId: 41 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41890 deadline: 1681527339542, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f8160290180ba126c904acc32b60eac7, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:39,544 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f8160290180ba126c904acc32b60eac7, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:54:39,544 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34681] ipc.CallRunner(144): callId: 42 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41918 deadline: 1681527339543, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f8160290180ba126c904acc32b60eac7, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:39,560 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=73 (bloomFilter=true), to=hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/f8160290180ba126c904acc32b60eac7/.tmp/A/86620e56e72f459a99b0d456ff3d5e39 2023-04-15 02:54:39,626 DEBUG [Thread-1527] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-15 02:54:39,626 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-15 02:54:39,628 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-15 02:54:39,629 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-15 02:54:39,629 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-15 02:54:39,629 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-15 02:54:39,629 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-15 02:54:39,629 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-15 02:54:39,629 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-15 02:54:39,629 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:39,629 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:54:39,630 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-15 02:54:39,630 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:39,630 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-15 02:54:39,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-15 02:54:39,630 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:39,630 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-15 02:54:39,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:54:39,630 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:39,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-15 02:54:39,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:54:39,631 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-15 02:54:39,631 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-15 02:54:39,631 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-15 02:54:39,631 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-15 02:54:39,631 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-15 02:54:39,631 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-15 02:54:39,631 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-15 02:54:39,631 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-15 02:54:39,631 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681527274083.f8160290180ba126c904acc32b60eac7. 2023-04-15 02:54:39,631 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681527274083.f8160290180ba126c904acc32b60eac7. started... 2023-04-15 02:54:39,631 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681527274083.f8160290180ba126c904acc32b60eac7. as already flushing 2023-04-15 02:54:39,632 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681527274083.f8160290180ba126c904acc32b60eac7. 2023-04-15 02:54:39,632 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-15 02:54:39,632 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-15 02:54:39,632 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:39,632 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-15 02:54:39,632 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34681,1681527209133' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-15 02:54:39,632 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:39,632 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:39,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:39,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:54:39,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:54:39,633 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:39,633 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-15 02:54:39,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:54:39,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:54:39,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:54:39,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:39,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:54:39,634 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34681,1681527209133' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-15 02:54:39,634 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-15 02:54:39,634 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@52598338[Count = 0] remaining members to acquire global barrier 2023-04-15 02:54:39,634 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:39,634 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:39,634 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:39,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:39,634 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-15 02:54:39,634 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-15 02:54:39,634 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:39,635 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-15 02:54:39,634 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34681,1681527209133' in zk 2023-04-15 02:54:39,636 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-15 02:54:39,636 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-15 02:54:39,636 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-15 02:54:39,638 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:39,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:39,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:54:39,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:54:39,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:54:39,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:54:39,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:54:39,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:39,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:54:39,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:54:39,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:39,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34681,1681527209133': 2023-04-15 02:54:39,641 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34681,1681527209133' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-15 02:54:39,641 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-15 02:54:39,641 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-15 02:54:39,641 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-15 02:54:39,641 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-15 02:54:39,641 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-15 02:54:39,650 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:39,650 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:39,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:39,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:54:39,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:54:39,650 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:39,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:39,650 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-15 02:54:39,651 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-15 02:54:39,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-15 02:54:39,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:54:39,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:39,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:54:39,652 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:39,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:54:39,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:54:39,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:39,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:54:39,653 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:39,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:54:39,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:39,658 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:39,658 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:54:39,658 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:54:39,658 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-15 02:54:39,658 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-15 02:54:39,658 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-15 02:54:39,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-15 02:54:39,658 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:39,658 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-15 02:54:39,659 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-15 02:54:39,658 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-15 02:54:39,658 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:39,659 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:39,659 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:39,659 DEBUG [Thread-1527] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-15 02:54:39,659 DEBUG [Thread-1527] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-15 02:54:39,659 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-15 02:54:39,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-15 02:54:39,759 DEBUG [Thread-1527] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-15 02:54:39,759 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-15 02:54:39,761 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-15 02:54:39,762 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-15 02:54:39,762 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-15 02:54:39,762 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-15 02:54:39,762 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-15 02:54:39,762 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-15 02:54:39,762 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-15 02:54:39,762 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:39,762 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:54:39,763 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:39,763 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-15 02:54:39,763 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-15 02:54:39,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-15 02:54:39,763 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:39,763 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-15 02:54:39,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:54:39,763 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:39,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-15 02:54:39,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:54:39,764 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-15 02:54:39,764 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-15 02:54:39,764 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-15 02:54:39,764 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-15 02:54:39,764 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-15 02:54:39,764 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-15 02:54:39,765 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-15 02:54:39,765 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-15 02:54:39,765 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681527274083.f8160290180ba126c904acc32b60eac7. 2023-04-15 02:54:39,765 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681527274083.f8160290180ba126c904acc32b60eac7. started... 2023-04-15 02:54:39,765 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681527274083.f8160290180ba126c904acc32b60eac7. as already flushing 2023-04-15 02:54:39,765 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681527274083.f8160290180ba126c904acc32b60eac7. 2023-04-15 02:54:39,765 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-15 02:54:39,765 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-15 02:54:39,765 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:39,765 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-15 02:54:39,765 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34681,1681527209133' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-15 02:54:39,766 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:39,766 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:39,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:39,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:54:39,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:54:39,766 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:39,766 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-15 02:54:39,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:54:39,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:54:39,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:54:39,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:39,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:54:39,767 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34681,1681527209133' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-15 02:54:39,767 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-15 02:54:39,767 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5e1a84d9[Count = 0] remaining members to acquire global barrier 2023-04-15 02:54:39,768 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:39,768 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:39,768 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:39,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:39,768 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-15 02:54:39,768 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-15 02:54:39,768 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34681,1681527209133' in zk 2023-04-15 02:54:39,769 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:39,769 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-15 02:54:39,769 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:39,769 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-15 02:54:39,769 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-15 02:54:39,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:39,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:54:39,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:54:39,769 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-15 02:54:39,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:54:39,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:54:39,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:54:39,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:39,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:54:39,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:54:39,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:39,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34681,1681527209133': 2023-04-15 02:54:39,771 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34681,1681527209133' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-15 02:54:39,771 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-15 02:54:39,771 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-15 02:54:39,771 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-15 02:54:39,771 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-15 02:54:39,771 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-15 02:54:39,772 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:39,772 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:39,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:39,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:54:39,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:54:39,772 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:39,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:39,772 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-15 02:54:39,772 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-15 02:54:39,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-15 02:54:39,772 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:39,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:54:39,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:39,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:54:39,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:54:39,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:54:39,773 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:39,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:39,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:54:39,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:54:39,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:39,775 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:39,775 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:54:39,775 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:54:39,775 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:39,775 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-15 02:54:39,775 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-15 02:54:39,775 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-15 02:54:39,775 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-15 02:54:39,775 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:39,775 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-15 02:54:39,775 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-15 02:54:39,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-15 02:54:39,775 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:39,775 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-15 02:54:39,775 DEBUG [Thread-1527] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-15 02:54:39,775 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:39,775 DEBUG [Thread-1527] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-15 02:54:39,775 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-15 02:54:39,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-15 02:54:39,779 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f8160290180ba126c904acc32b60eac7, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:54:39,779 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34681] ipc.CallRunner(144): callId: 37 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41916 deadline: 1681527339778, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f8160290180ba126c904acc32b60eac7, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:39,784 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f8160290180ba126c904acc32b60eac7, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:54:39,784 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34681] ipc.CallRunner(144): callId: 38 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41940 deadline: 1681527339782, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f8160290180ba126c904acc32b60eac7, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:39,842 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f8160290180ba126c904acc32b60eac7, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:54:39,842 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34681] ipc.CallRunner(144): callId: 47 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41926 deadline: 1681527339840, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f8160290180ba126c904acc32b60eac7, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:39,844 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f8160290180ba126c904acc32b60eac7, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:54:39,845 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34681] ipc.CallRunner(144): callId: 43 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41890 deadline: 1681527339844, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f8160290180ba126c904acc32b60eac7, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:39,847 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f8160290180ba126c904acc32b60eac7, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:54:39,847 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34681] ipc.CallRunner(144): callId: 44 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41918 deadline: 1681527339846, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f8160290180ba126c904acc32b60eac7, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:39,875 DEBUG [Thread-1527] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-15 02:54:39,876 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-15 02:54:39,877 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-15 02:54:39,878 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-15 02:54:39,878 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-15 02:54:39,878 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-15 02:54:39,878 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-15 02:54:39,878 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-15 02:54:39,878 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-15 02:54:39,878 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:39,878 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:54:39,879 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-15 02:54:39,879 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:39,879 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-15 02:54:39,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-15 02:54:39,879 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:39,879 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-15 02:54:39,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:54:39,880 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:39,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-15 02:54:39,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:54:39,880 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-15 02:54:39,881 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-15 02:54:39,881 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-15 02:54:39,881 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-15 02:54:39,881 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-15 02:54:39,881 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-15 02:54:39,881 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-15 02:54:39,881 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-15 02:54:39,881 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681527274083.f8160290180ba126c904acc32b60eac7. 2023-04-15 02:54:39,881 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681527274083.f8160290180ba126c904acc32b60eac7. started... 2023-04-15 02:54:39,881 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681527274083.f8160290180ba126c904acc32b60eac7. as already flushing 2023-04-15 02:54:39,881 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681527274083.f8160290180ba126c904acc32b60eac7. 2023-04-15 02:54:39,881 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-15 02:54:39,881 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-15 02:54:39,881 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:39,881 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-15 02:54:39,882 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34681,1681527209133' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-15 02:54:39,882 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:39,882 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:39,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:39,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:54:39,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:54:39,883 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:39,883 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-15 02:54:39,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:54:39,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:54:39,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:54:39,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:39,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:54:39,884 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34681,1681527209133' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-15 02:54:39,884 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-15 02:54:39,884 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@16b3efbe[Count = 0] remaining members to acquire global barrier 2023-04-15 02:54:39,884 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:39,884 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:39,884 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:39,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:39,884 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-15 02:54:39,884 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:39,884 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-15 02:54:39,884 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-15 02:54:39,884 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34681,1681527209133' in zk 2023-04-15 02:54:39,885 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:39,885 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-15 02:54:39,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:39,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:54:39,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:54:39,885 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-15 02:54:39,885 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-15 02:54:39,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:54:39,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:54:39,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:54:39,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:39,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:54:39,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:54:39,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:39,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34681,1681527209133': 2023-04-15 02:54:39,887 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34681,1681527209133' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-15 02:54:39,887 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-15 02:54:39,887 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-15 02:54:39,887 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-15 02:54:39,887 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-15 02:54:39,887 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-15 02:54:39,888 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:39,888 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:39,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:39,888 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:39,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:39,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:54:39,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:54:39,888 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-15 02:54:39,888 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:39,888 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-15 02:54:39,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-15 02:54:39,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:39,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:54:39,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:54:39,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:54:39,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:54:39,890 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:39,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:39,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:54:39,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:54:39,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:39,891 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:39,891 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:54:39,891 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-15 02:54:39,891 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:54:39,891 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-15 02:54:39,891 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-15 02:54:39,891 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-15 02:54:39,891 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-15 02:54:39,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-15 02:54:39,891 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-15 02:54:39,891 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-15 02:54:39,891 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:39,892 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:39,892 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-15 02:54:39,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-15 02:54:39,892 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:39,892 DEBUG [Thread-1527] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-15 02:54:39,892 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:39,892 DEBUG [Thread-1527] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-15 02:54:39,980 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=73 (bloomFilter=true), to=hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/f8160290180ba126c904acc32b60eac7/.tmp/B/0f93ea44f44f4b189b1a30893dbf6367 2023-04-15 02:54:39,992 DEBUG [Thread-1527] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-15 02:54:39,992 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-15 02:54:39,993 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-15 02:54:39,994 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-15 02:54:39,994 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-15 02:54:39,994 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-15 02:54:39,994 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-15 02:54:39,994 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-15 02:54:39,994 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-15 02:54:39,995 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:39,995 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:54:39,995 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-15 02:54:39,995 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-15 02:54:39,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-15 02:54:39,995 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:39,996 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:39,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:54:39,996 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-15 02:54:39,996 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:39,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-15 02:54:39,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:54:39,996 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-15 02:54:39,996 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-15 02:54:39,996 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-15 02:54:39,996 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-15 02:54:39,996 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-15 02:54:39,996 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-15 02:54:39,997 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-15 02:54:39,997 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681527274083.f8160290180ba126c904acc32b60eac7. 2023-04-15 02:54:39,997 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681527274083.f8160290180ba126c904acc32b60eac7. started... 2023-04-15 02:54:39,997 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-15 02:54:39,997 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681527274083.f8160290180ba126c904acc32b60eac7. as already flushing 2023-04-15 02:54:39,997 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681527274083.f8160290180ba126c904acc32b60eac7. 2023-04-15 02:54:39,997 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-15 02:54:39,997 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-15 02:54:39,997 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:39,997 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-15 02:54:39,997 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34681,1681527209133' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-15 02:54:39,998 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:39,998 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:39,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:39,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:54:39,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:54:39,999 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=73 (bloomFilter=true), to=hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/f8160290180ba126c904acc32b60eac7/.tmp/C/112f7d90f0894cf7b4f86b1034fb5a3c 2023-04-15 02:54:40,002 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:40,002 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-15 02:54:40,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:54:40,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:54:40,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:54:40,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:40,006 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/f8160290180ba126c904acc32b60eac7/.tmp/A/86620e56e72f459a99b0d456ff3d5e39 as hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/f8160290180ba126c904acc32b60eac7/A/86620e56e72f459a99b0d456ff3d5e39 2023-04-15 02:54:40,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:54:40,007 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34681,1681527209133' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-15 02:54:40,007 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@a3419a0[Count = 0] remaining members to acquire global barrier 2023-04-15 02:54:40,007 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-15 02:54:40,008 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:40,010 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/f8160290180ba126c904acc32b60eac7/A/86620e56e72f459a99b0d456ff3d5e39, entries=250, sequenceid=73, filesize=16.2 K 2023-04-15 02:54:40,010 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/f8160290180ba126c904acc32b60eac7/.tmp/B/0f93ea44f44f4b189b1a30893dbf6367 as hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/f8160290180ba126c904acc32b60eac7/B/0f93ea44f44f4b189b1a30893dbf6367 2023-04-15 02:54:40,012 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:40,012 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:40,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:40,012 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:40,012 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-15 02:54:40,012 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-15 02:54:40,012 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-15 02:54:40,012 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34681,1681527209133' in zk 2023-04-15 02:54:40,018 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-15 02:54:40,018 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:40,018 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-15 02:54:40,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:40,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:54:40,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:54:40,018 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-15 02:54:40,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:54:40,018 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/f8160290180ba126c904acc32b60eac7/B/0f93ea44f44f4b189b1a30893dbf6367, entries=150, sequenceid=73, filesize=11.6 K 2023-04-15 02:54:40,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:54:40,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:54:40,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:40,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:54:40,019 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/f8160290180ba126c904acc32b60eac7/.tmp/C/112f7d90f0894cf7b4f86b1034fb5a3c as hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/f8160290180ba126c904acc32b60eac7/C/112f7d90f0894cf7b4f86b1034fb5a3c 2023-04-15 02:54:40,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:54:40,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:40,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34681,1681527209133': 2023-04-15 02:54:40,020 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34681,1681527209133' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-15 02:54:40,020 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-15 02:54:40,020 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-15 02:54:40,020 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-15 02:54:40,020 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-15 02:54:40,020 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-15 02:54:40,022 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/f8160290180ba126c904acc32b60eac7/C/112f7d90f0894cf7b4f86b1034fb5a3c, entries=150, sequenceid=73, filesize=11.6 K 2023-04-15 02:54:40,023 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=129.71 KB/132820 for f8160290180ba126c904acc32b60eac7 in 879ms, sequenceid=73, compaction requested=true 2023-04-15 02:54:40,023 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for f8160290180ba126c904acc32b60eac7: 2023-04-15 02:54:40,023 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-15 02:54:40,023 DEBUG [RS:0;jenkins-hbase20:34681-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-15 02:54:40,023 DEBUG [RS:0;jenkins-hbase20:34681-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-15 02:54:40,023 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-15 02:54:40,023 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-15 02:54:40,026 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:40,026 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:40,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:40,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:54:40,026 DEBUG [RS:0;jenkins-hbase20:34681-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 52224 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-15 02:54:40,026 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:40,026 DEBUG [RS:0;jenkins-hbase20:34681-shortCompactions-0] regionserver.HStore(1912): f8160290180ba126c904acc32b60eac7/A is initiating minor compaction (all files) 2023-04-15 02:54:40,026 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-15 02:54:40,026 INFO [RS:0;jenkins-hbase20:34681-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of f8160290180ba126c904acc32b60eac7/A in TestAcidGuarantees,,1681527274083.f8160290180ba126c904acc32b60eac7. 2023-04-15 02:54:40,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:40,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:54:40,026 DEBUG [RS:0;jenkins-hbase20:34681-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 47544 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-15 02:54:40,026 INFO [RS:0;jenkins-hbase20:34681-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/f8160290180ba126c904acc32b60eac7/A/40f07466e76f415ab019342c8727995e, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/f8160290180ba126c904acc32b60eac7/A/9d601bd6d01f4a3f808814012e1a1b12, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/f8160290180ba126c904acc32b60eac7/A/f4e10dcc69934f22941dd7fde1d40ee1, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/f8160290180ba126c904acc32b60eac7/A/86620e56e72f459a99b0d456ff3d5e39] into tmpdir=hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/f8160290180ba126c904acc32b60eac7/.tmp, totalSize=51 K 2023-04-15 02:54:40,026 DEBUG [RS:0;jenkins-hbase20:34681-longCompactions-0] regionserver.HStore(1912): f8160290180ba126c904acc32b60eac7/B is initiating minor compaction (all files) 2023-04-15 02:54:40,026 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:40,026 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-15 02:54:40,026 INFO [RS:0;jenkins-hbase20:34681-longCompactions-0] regionserver.HRegion(2259): Starting compaction of f8160290180ba126c904acc32b60eac7/B in TestAcidGuarantees,,1681527274083.f8160290180ba126c904acc32b60eac7. 2023-04-15 02:54:40,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-15 02:54:40,027 INFO [RS:0;jenkins-hbase20:34681-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/f8160290180ba126c904acc32b60eac7/B/b8c69868004c408eb9d6fd873d117041, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/f8160290180ba126c904acc32b60eac7/B/abc1edeb35f04e01b9b1176e384165f7, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/f8160290180ba126c904acc32b60eac7/B/e8ba002fe0f945c0b9561495f7b6e3a5, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/f8160290180ba126c904acc32b60eac7/B/0f93ea44f44f4b189b1a30893dbf6367] into tmpdir=hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/f8160290180ba126c904acc32b60eac7/.tmp, totalSize=46.4 K 2023-04-15 02:54:40,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:54:40,027 DEBUG [RS:0;jenkins-hbase20:34681-shortCompactions-0] compactions.Compactor(207): Compacting 40f07466e76f415ab019342c8727995e, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=12, earliestPutTs=1681527276307 2023-04-15 02:54:40,027 DEBUG [RS:0;jenkins-hbase20:34681-longCompactions-0] compactions.Compactor(207): Compacting b8c69868004c408eb9d6fd873d117041, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=12, earliestPutTs=1681527276307 2023-04-15 02:54:40,027 DEBUG [RS:0;jenkins-hbase20:34681-shortCompactions-0] compactions.Compactor(207): Compacting 9d601bd6d01f4a3f808814012e1a1b12, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=47, earliestPutTs=1681527276351 2023-04-15 02:54:40,027 DEBUG [RS:0;jenkins-hbase20:34681-longCompactions-0] compactions.Compactor(207): Compacting abc1edeb35f04e01b9b1176e384165f7, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=47, earliestPutTs=1681527276351 2023-04-15 02:54:40,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:40,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:54:40,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:54:40,030 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:40,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:54:40,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:40,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:54:40,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:54:40,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:40,032 DEBUG [RS:0;jenkins-hbase20:34681-shortCompactions-0] compactions.Compactor(207): Compacting f4e10dcc69934f22941dd7fde1d40ee1, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=59, earliestPutTs=1681527276991 2023-04-15 02:54:40,032 DEBUG [RS:0;jenkins-hbase20:34681-longCompactions-0] compactions.Compactor(207): Compacting e8ba002fe0f945c0b9561495f7b6e3a5, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=59, earliestPutTs=1681527276991 2023-04-15 02:54:40,033 DEBUG [RS:0;jenkins-hbase20:34681-shortCompactions-0] compactions.Compactor(207): Compacting 86620e56e72f459a99b0d456ff3d5e39, keycount=250, bloomtype=ROW, size=16.2 K, encoding=NONE, compression=NONE, seqNum=73, earliestPutTs=1681527279139 2023-04-15 02:54:40,033 DEBUG [RS:0;jenkins-hbase20:34681-longCompactions-0] compactions.Compactor(207): Compacting 0f93ea44f44f4b189b1a30893dbf6367, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=73, earliestPutTs=1681527279139 2023-04-15 02:54:40,033 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:40,033 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:54:40,033 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:54:40,033 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-15 02:54:40,033 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-15 02:54:40,033 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:40,033 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-15 02:54:40,033 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:40,033 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-15 02:54:40,033 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-15 02:54:40,033 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-15 02:54:40,033 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:40,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-15 02:54:40,033 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-15 02:54:40,033 DEBUG [Thread-1527] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-15 02:54:40,034 DEBUG [Thread-1527] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-15 02:54:40,034 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-15 02:54:40,033 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:40,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-15 02:54:40,041 INFO [RS:0;jenkins-hbase20:34681-shortCompactions-0] throttle.PressureAwareThroughputController(145): f8160290180ba126c904acc32b60eac7#A#compaction#191 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-15 02:54:40,042 INFO [RS:0;jenkins-hbase20:34681-longCompactions-0] throttle.PressureAwareThroughputController(145): f8160290180ba126c904acc32b60eac7#B#compaction#192 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-15 02:54:40,056 DEBUG [RS:0;jenkins-hbase20:34681-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/f8160290180ba126c904acc32b60eac7/.tmp/A/61dcb71e9705420ab52f98869f5eab5a as hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/f8160290180ba126c904acc32b60eac7/A/61dcb71e9705420ab52f98869f5eab5a 2023-04-15 02:54:40,060 INFO [RS:0;jenkins-hbase20:34681-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in f8160290180ba126c904acc32b60eac7/A of f8160290180ba126c904acc32b60eac7 into 61dcb71e9705420ab52f98869f5eab5a(size=11.7 K), total size for store is 11.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-15 02:54:40,060 DEBUG [RS:0;jenkins-hbase20:34681-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for f8160290180ba126c904acc32b60eac7: 2023-04-15 02:54:40,060 INFO [RS:0;jenkins-hbase20:34681-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681527274083.f8160290180ba126c904acc32b60eac7., storeName=f8160290180ba126c904acc32b60eac7/A, priority=12, startTime=1681527280023; duration=0sec 2023-04-15 02:54:40,060 DEBUG [RS:0;jenkins-hbase20:34681-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-15 02:54:40,060 DEBUG [RS:0;jenkins-hbase20:34681-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-15 02:54:40,061 DEBUG [RS:0;jenkins-hbase20:34681-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 47544 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-15 02:54:40,061 DEBUG [RS:0;jenkins-hbase20:34681-shortCompactions-0] regionserver.HStore(1912): f8160290180ba126c904acc32b60eac7/C is initiating minor compaction (all files) 2023-04-15 02:54:40,061 INFO [RS:0;jenkins-hbase20:34681-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of f8160290180ba126c904acc32b60eac7/C in TestAcidGuarantees,,1681527274083.f8160290180ba126c904acc32b60eac7. 2023-04-15 02:54:40,061 INFO [RS:0;jenkins-hbase20:34681-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/f8160290180ba126c904acc32b60eac7/C/925c7d2aa44e4aa1b8865d4572d399ed, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/f8160290180ba126c904acc32b60eac7/C/1c397bc1e228408e8cd10a50d92af44f, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/f8160290180ba126c904acc32b60eac7/C/f4d03398fbf44a50944d5ba548a577d2, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/f8160290180ba126c904acc32b60eac7/C/112f7d90f0894cf7b4f86b1034fb5a3c] into tmpdir=hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/f8160290180ba126c904acc32b60eac7/.tmp, totalSize=46.4 K 2023-04-15 02:54:40,062 DEBUG [RS:0;jenkins-hbase20:34681-shortCompactions-0] compactions.Compactor(207): Compacting 925c7d2aa44e4aa1b8865d4572d399ed, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=12, earliestPutTs=1681527276307 2023-04-15 02:54:40,062 DEBUG [RS:0;jenkins-hbase20:34681-shortCompactions-0] compactions.Compactor(207): Compacting 1c397bc1e228408e8cd10a50d92af44f, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=47, earliestPutTs=1681527276351 2023-04-15 02:54:40,062 DEBUG [RS:0;jenkins-hbase20:34681-shortCompactions-0] compactions.Compactor(207): Compacting f4d03398fbf44a50944d5ba548a577d2, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=59, earliestPutTs=1681527276991 2023-04-15 02:54:40,063 DEBUG [RS:0;jenkins-hbase20:34681-shortCompactions-0] compactions.Compactor(207): Compacting 112f7d90f0894cf7b4f86b1034fb5a3c, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=73, earliestPutTs=1681527279139 2023-04-15 02:54:40,068 INFO [RS:0;jenkins-hbase20:34681-shortCompactions-0] throttle.PressureAwareThroughputController(145): f8160290180ba126c904acc32b60eac7#C#compaction#193 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-15 02:54:40,082 DEBUG [RS:0;jenkins-hbase20:34681-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/f8160290180ba126c904acc32b60eac7/.tmp/C/29656fb2fd6c4e998f291542df8d67fc as hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/f8160290180ba126c904acc32b60eac7/C/29656fb2fd6c4e998f291542df8d67fc 2023-04-15 02:54:40,086 INFO [RS:0;jenkins-hbase20:34681-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in f8160290180ba126c904acc32b60eac7/C of f8160290180ba126c904acc32b60eac7 into 29656fb2fd6c4e998f291542df8d67fc(size=11.7 K), total size for store is 11.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-15 02:54:40,086 DEBUG [RS:0;jenkins-hbase20:34681-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for f8160290180ba126c904acc32b60eac7: 2023-04-15 02:54:40,086 INFO [RS:0;jenkins-hbase20:34681-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681527274083.f8160290180ba126c904acc32b60eac7., storeName=f8160290180ba126c904acc32b60eac7/C, priority=12, startTime=1681527280023; duration=0sec 2023-04-15 02:54:40,086 DEBUG [RS:0;jenkins-hbase20:34681-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-15 02:54:40,134 DEBUG [Thread-1527] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-15 02:54:40,134 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-15 02:54:40,136 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-15 02:54:40,137 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-15 02:54:40,137 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-15 02:54:40,137 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-15 02:54:40,137 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-15 02:54:40,137 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-15 02:54:40,137 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-15 02:54:40,137 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:40,138 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:54:40,138 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:40,138 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-15 02:54:40,138 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-15 02:54:40,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-15 02:54:40,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:54:40,140 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:40,140 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-15 02:54:40,140 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:40,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-15 02:54:40,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:54:40,141 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-15 02:54:40,141 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-15 02:54:40,141 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-15 02:54:40,141 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-15 02:54:40,141 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-15 02:54:40,141 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-15 02:54:40,141 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-15 02:54:40,141 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-15 02:54:40,141 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681527274083.f8160290180ba126c904acc32b60eac7. 2023-04-15 02:54:40,142 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681527274083.f8160290180ba126c904acc32b60eac7. started... 2023-04-15 02:54:40,142 INFO [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing f8160290180ba126c904acc32b60eac7 3/3 column families, dataSize=129.71 KB heapSize=365.45 KB 2023-04-15 02:54:40,143 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK f8160290180ba126c904acc32b60eac7, store=A 2023-04-15 02:54:40,143 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-15 02:54:40,143 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK f8160290180ba126c904acc32b60eac7, store=B 2023-04-15 02:54:40,143 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-15 02:54:40,143 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK f8160290180ba126c904acc32b60eac7, store=C 2023-04-15 02:54:40,143 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-15 02:54:40,177 INFO [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=110 (bloomFilter=true), to=hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/f8160290180ba126c904acc32b60eac7/.tmp/A/0944ea0d48e048fe8486e8aa032e1470 2023-04-15 02:54:40,283 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34681] regionserver.HRegion(9158): Flush requested on f8160290180ba126c904acc32b60eac7 2023-04-15 02:54:40,283 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681527274083.f8160290180ba126c904acc32b60eac7. as already flushing 2023-04-15 02:54:40,299 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f8160290180ba126c904acc32b60eac7, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:54:40,299 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34681] ipc.CallRunner(144): callId: 44 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41916 deadline: 1681527340296, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f8160290180ba126c904acc32b60eac7, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:40,301 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f8160290180ba126c904acc32b60eac7, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:54:40,301 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34681] ipc.CallRunner(144): callId: 43 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41940 deadline: 1681527340299, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f8160290180ba126c904acc32b60eac7, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:40,345 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f8160290180ba126c904acc32b60eac7, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:54:40,345 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34681] ipc.CallRunner(144): callId: 49 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41926 deadline: 1681527340345, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f8160290180ba126c904acc32b60eac7, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:40,347 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f8160290180ba126c904acc32b60eac7, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:54:40,348 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34681] ipc.CallRunner(144): callId: 45 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41890 deadline: 1681527340345, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f8160290180ba126c904acc32b60eac7, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:40,351 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f8160290180ba126c904acc32b60eac7, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:54:40,351 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34681] ipc.CallRunner(144): callId: 46 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41918 deadline: 1681527340350, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f8160290180ba126c904acc32b60eac7, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:40,401 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f8160290180ba126c904acc32b60eac7, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:54:40,401 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34681] ipc.CallRunner(144): callId: 46 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41916 deadline: 1681527340400, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f8160290180ba126c904acc32b60eac7, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:40,403 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f8160290180ba126c904acc32b60eac7, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:54:40,404 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34681] ipc.CallRunner(144): callId: 45 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41940 deadline: 1681527340402, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f8160290180ba126c904acc32b60eac7, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:40,455 DEBUG [RS:0;jenkins-hbase20:34681-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/f8160290180ba126c904acc32b60eac7/.tmp/B/2e47e14a986e4838b5afda846a0524da as hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/f8160290180ba126c904acc32b60eac7/B/2e47e14a986e4838b5afda846a0524da 2023-04-15 02:54:40,461 INFO [RS:0;jenkins-hbase20:34681-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in f8160290180ba126c904acc32b60eac7/B of f8160290180ba126c904acc32b60eac7 into 2e47e14a986e4838b5afda846a0524da(size=11.7 K), total size for store is 11.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-15 02:54:40,461 DEBUG [RS:0;jenkins-hbase20:34681-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for f8160290180ba126c904acc32b60eac7: 2023-04-15 02:54:40,461 INFO [RS:0;jenkins-hbase20:34681-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681527274083.f8160290180ba126c904acc32b60eac7., storeName=f8160290180ba126c904acc32b60eac7/B, priority=12, startTime=1681527280023; duration=0sec 2023-04-15 02:54:40,461 DEBUG [RS:0;jenkins-hbase20:34681-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-15 02:54:40,606 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f8160290180ba126c904acc32b60eac7, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:54:40,606 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34681] ipc.CallRunner(144): callId: 47 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41940 deadline: 1681527340605, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f8160290180ba126c904acc32b60eac7, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:40,607 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f8160290180ba126c904acc32b60eac7, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:54:40,607 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34681] ipc.CallRunner(144): callId: 48 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41916 deadline: 1681527340606, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f8160290180ba126c904acc32b60eac7, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:40,640 INFO [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=110 (bloomFilter=true), to=hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/f8160290180ba126c904acc32b60eac7/.tmp/B/0b2852aa58d9444994a5a57c93b0d068 2023-04-15 02:54:40,909 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f8160290180ba126c904acc32b60eac7, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:54:40,909 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34681] ipc.CallRunner(144): callId: 49 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41940 deadline: 1681527340908, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f8160290180ba126c904acc32b60eac7, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:40,911 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f8160290180ba126c904acc32b60eac7, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:54:40,911 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34681] ipc.CallRunner(144): callId: 50 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41916 deadline: 1681527340909, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f8160290180ba126c904acc32b60eac7, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:41,067 INFO [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=110 (bloomFilter=true), to=hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/f8160290180ba126c904acc32b60eac7/.tmp/C/84dfea7961624d9baeee6e9124300099 2023-04-15 02:54:41,071 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/f8160290180ba126c904acc32b60eac7/.tmp/A/0944ea0d48e048fe8486e8aa032e1470 as hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/f8160290180ba126c904acc32b60eac7/A/0944ea0d48e048fe8486e8aa032e1470 2023-04-15 02:54:41,075 INFO [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/f8160290180ba126c904acc32b60eac7/A/0944ea0d48e048fe8486e8aa032e1470, entries=150, sequenceid=110, filesize=11.6 K 2023-04-15 02:54:41,075 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/f8160290180ba126c904acc32b60eac7/.tmp/B/0b2852aa58d9444994a5a57c93b0d068 as hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/f8160290180ba126c904acc32b60eac7/B/0b2852aa58d9444994a5a57c93b0d068 2023-04-15 02:54:41,079 INFO [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/f8160290180ba126c904acc32b60eac7/B/0b2852aa58d9444994a5a57c93b0d068, entries=150, sequenceid=110, filesize=11.6 K 2023-04-15 02:54:41,080 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/f8160290180ba126c904acc32b60eac7/.tmp/C/84dfea7961624d9baeee6e9124300099 as hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/f8160290180ba126c904acc32b60eac7/C/84dfea7961624d9baeee6e9124300099 2023-04-15 02:54:41,083 INFO [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/f8160290180ba126c904acc32b60eac7/C/84dfea7961624d9baeee6e9124300099, entries=150, sequenceid=110, filesize=11.6 K 2023-04-15 02:54:41,084 INFO [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~129.71 KB/132820, heapSize ~365.41 KB/374176, currentSize=53.67 KB/54960 for f8160290180ba126c904acc32b60eac7 in 942ms, sequenceid=110, compaction requested=false 2023-04-15 02:54:41,084 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for f8160290180ba126c904acc32b60eac7: 2023-04-15 02:54:41,084 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681527274083.f8160290180ba126c904acc32b60eac7. 2023-04-15 02:54:41,084 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-15 02:54:41,084 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-15 02:54:41,084 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:41,084 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-15 02:54:41,084 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34681,1681527209133' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-15 02:54:41,086 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:41,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:41,086 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:41,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:54:41,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:54:41,086 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:41,086 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-15 02:54:41,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:54:41,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:54:41,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:54:41,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:41,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:54:41,087 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34681,1681527209133' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-15 02:54:41,088 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@14ea1b9d[Count = 0] remaining members to acquire global barrier 2023-04-15 02:54:41,088 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-15 02:54:41,088 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:41,088 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:41,088 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:41,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:41,088 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-15 02:54:41,088 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-15 02:54:41,088 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34681,1681527209133' in zk 2023-04-15 02:54:41,089 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:41,089 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-15 02:54:41,089 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:41,089 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-15 02:54:41,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:41,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:54:41,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:54:41,089 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-15 02:54:41,089 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-15 02:54:41,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:54:41,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:54:41,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:54:41,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:41,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:54:41,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:54:41,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:41,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34681,1681527209133': 2023-04-15 02:54:41,092 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34681,1681527209133' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-15 02:54:41,092 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-15 02:54:41,093 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-15 02:54:41,093 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-15 02:54:41,093 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-15 02:54:41,093 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-15 02:54:41,103 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:41,103 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:41,103 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:41,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:41,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:54:41,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:54:41,103 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-15 02:54:41,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:41,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:54:41,104 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:41,104 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-15 02:54:41,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-15 02:54:41,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:54:41,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:41,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:54:41,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:54:41,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:41,106 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:41,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:54:41,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:54:41,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:41,108 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-15 02:54:41,108 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-15 02:54:41,108 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:41,108 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:54:41,108 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:54:41,108 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:41,108 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:41,108 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:41,108 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:41,108 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-15 02:54:41,108 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-15 02:54:41,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-15 02:54:41,108 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-15 02:54:41,108 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-15 02:54:41,108 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-15 02:54:41,108 DEBUG [Thread-1527] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-15 02:54:41,108 DEBUG [Thread-1527] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-15 02:54:41,108 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-15 02:54:41,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-15 02:54:41,208 DEBUG [Thread-1527] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-15 02:54:41,209 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-15 02:54:41,211 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-15 02:54:41,212 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-15 02:54:41,212 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-15 02:54:41,212 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-15 02:54:41,212 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-15 02:54:41,212 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-15 02:54:41,212 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-15 02:54:41,212 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:41,212 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:54:41,213 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:41,213 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-15 02:54:41,213 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-15 02:54:41,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-15 02:54:41,213 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:41,213 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-15 02:54:41,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:54:41,213 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:41,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-15 02:54:41,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:54:41,214 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-15 02:54:41,214 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-15 02:54:41,214 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-15 02:54:41,214 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-15 02:54:41,214 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-15 02:54:41,214 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-15 02:54:41,218 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-15 02:54:41,218 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681527274083.f8160290180ba126c904acc32b60eac7. 2023-04-15 02:54:41,218 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-15 02:54:41,218 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681527274083.f8160290180ba126c904acc32b60eac7. started... 2023-04-15 02:54:41,218 INFO [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing f8160290180ba126c904acc32b60eac7 3/3 column families, dataSize=53.67 KB heapSize=160.13 KB 2023-04-15 02:54:41,218 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK f8160290180ba126c904acc32b60eac7, store=A 2023-04-15 02:54:41,218 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-15 02:54:41,219 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK f8160290180ba126c904acc32b60eac7, store=B 2023-04-15 02:54:41,219 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-15 02:54:41,219 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK f8160290180ba126c904acc32b60eac7, store=C 2023-04-15 02:54:41,219 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-15 02:54:41,248 INFO [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=122 (bloomFilter=true), to=hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/f8160290180ba126c904acc32b60eac7/.tmp/A/3252d9d39f894e3e98cd1a5aa3e3014d 2023-04-15 02:54:41,353 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34681] regionserver.HRegion(9158): Flush requested on f8160290180ba126c904acc32b60eac7 2023-04-15 02:54:41,354 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681527274083.f8160290180ba126c904acc32b60eac7. as already flushing 2023-04-15 02:54:41,373 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1681527281360, max timestamp=1681527281371Num uniques 859; 2023-04-15 02:54:41,373 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f8160290180ba126c904acc32b60eac7, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:54:41,373 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-3] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-15 02:54:41,373 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34681] ipc.CallRunner(144): callId: 55 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41918 deadline: 1681527341370, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f8160290180ba126c904acc32b60eac7, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:41,375 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1681527281360, max timestamp=1681527281371Num uniques 859; 2023-04-15 02:54:41,375 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-8] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-15 02:54:41,375 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1681527281360, max timestamp=1681527281371Num uniques 859; 2023-04-15 02:54:41,375 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-3] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-15 02:54:41,418 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f8160290180ba126c904acc32b60eac7, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:54:41,418 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34681] ipc.CallRunner(144): callId: 52 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41940 deadline: 1681527341414, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f8160290180ba126c904acc32b60eac7, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:41,420 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f8160290180ba126c904acc32b60eac7, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:54:41,420 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34681] ipc.CallRunner(144): callId: 59 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41890 deadline: 1681527341417, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f8160290180ba126c904acc32b60eac7, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:41,422 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f8160290180ba126c904acc32b60eac7, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:54:41,422 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f8160290180ba126c904acc32b60eac7, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:54:41,422 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34681] ipc.CallRunner(144): callId: 53 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41916 deadline: 1681527341418, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f8160290180ba126c904acc32b60eac7, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:41,422 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34681] ipc.CallRunner(144): callId: 63 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41926 deadline: 1681527341418, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f8160290180ba126c904acc32b60eac7, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:41,476 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f8160290180ba126c904acc32b60eac7, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:54:41,476 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34681] ipc.CallRunner(144): callId: 57 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41918 deadline: 1681527341474, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f8160290180ba126c904acc32b60eac7, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:41,520 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f8160290180ba126c904acc32b60eac7, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:54:41,520 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34681] ipc.CallRunner(144): callId: 54 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41940 deadline: 1681527341519, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f8160290180ba126c904acc32b60eac7, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:41,522 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f8160290180ba126c904acc32b60eac7, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:54:41,522 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34681] ipc.CallRunner(144): callId: 61 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41890 deadline: 1681527341521, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f8160290180ba126c904acc32b60eac7, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:41,526 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f8160290180ba126c904acc32b60eac7, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:54:41,526 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34681] ipc.CallRunner(144): callId: 55 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41916 deadline: 1681527341523, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f8160290180ba126c904acc32b60eac7, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:41,526 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f8160290180ba126c904acc32b60eac7, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:54:41,526 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34681] ipc.CallRunner(144): callId: 65 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41926 deadline: 1681527341523, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f8160290180ba126c904acc32b60eac7, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:41,668 INFO [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=122 (bloomFilter=true), to=hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/f8160290180ba126c904acc32b60eac7/.tmp/B/b279733ba1f64aa5af8bfe1a1e7e0cb5 2023-04-15 02:54:41,678 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f8160290180ba126c904acc32b60eac7, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:54:41,679 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34681] ipc.CallRunner(144): callId: 59 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41918 deadline: 1681527341677, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f8160290180ba126c904acc32b60eac7, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:41,708 INFO [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=122 (bloomFilter=true), to=hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/f8160290180ba126c904acc32b60eac7/.tmp/C/9cd152c0805f49fdace043f6485e1918 2023-04-15 02:54:41,715 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/f8160290180ba126c904acc32b60eac7/.tmp/A/3252d9d39f894e3e98cd1a5aa3e3014d as hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/f8160290180ba126c904acc32b60eac7/A/3252d9d39f894e3e98cd1a5aa3e3014d 2023-04-15 02:54:41,721 INFO [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/f8160290180ba126c904acc32b60eac7/A/3252d9d39f894e3e98cd1a5aa3e3014d, entries=150, sequenceid=122, filesize=11.6 K 2023-04-15 02:54:41,723 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/f8160290180ba126c904acc32b60eac7/.tmp/B/b279733ba1f64aa5af8bfe1a1e7e0cb5 as hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/f8160290180ba126c904acc32b60eac7/B/b279733ba1f64aa5af8bfe1a1e7e0cb5 2023-04-15 02:54:41,724 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f8160290180ba126c904acc32b60eac7, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:54:41,724 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34681] ipc.CallRunner(144): callId: 56 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41940 deadline: 1681527341722, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f8160290180ba126c904acc32b60eac7, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:41,740 INFO [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/f8160290180ba126c904acc32b60eac7/B/b279733ba1f64aa5af8bfe1a1e7e0cb5, entries=150, sequenceid=122, filesize=11.6 K 2023-04-15 02:54:41,741 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f8160290180ba126c904acc32b60eac7, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:54:41,741 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34681] ipc.CallRunner(144): callId: 63 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41890 deadline: 1681527341725, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f8160290180ba126c904acc32b60eac7, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:41,741 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/f8160290180ba126c904acc32b60eac7/.tmp/C/9cd152c0805f49fdace043f6485e1918 as hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/f8160290180ba126c904acc32b60eac7/C/9cd152c0805f49fdace043f6485e1918 2023-04-15 02:54:41,741 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f8160290180ba126c904acc32b60eac7, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:54:41,742 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34681] ipc.CallRunner(144): callId: 57 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41916 deadline: 1681527341730, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f8160290180ba126c904acc32b60eac7, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:41,749 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f8160290180ba126c904acc32b60eac7, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:54:41,751 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34681] ipc.CallRunner(144): callId: 67 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41926 deadline: 1681527341730, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f8160290180ba126c904acc32b60eac7, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:41,768 INFO [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/f8160290180ba126c904acc32b60eac7/C/9cd152c0805f49fdace043f6485e1918, entries=150, sequenceid=122, filesize=11.6 K 2023-04-15 02:54:41,769 INFO [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=134.18 KB/137400 for f8160290180ba126c904acc32b60eac7 in 551ms, sequenceid=122, compaction requested=true 2023-04-15 02:54:41,769 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for f8160290180ba126c904acc32b60eac7: 2023-04-15 02:54:41,769 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681527274083.f8160290180ba126c904acc32b60eac7. 2023-04-15 02:54:41,769 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-15 02:54:41,769 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-15 02:54:41,769 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:41,770 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-15 02:54:41,770 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34681,1681527209133' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-15 02:54:41,771 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:41,771 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:41,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:41,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:54:41,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:54:41,771 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:41,771 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-15 02:54:41,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:54:41,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:54:41,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:54:41,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:41,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:54:41,774 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34681,1681527209133' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-15 02:54:41,774 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1e5c7995[Count = 0] remaining members to acquire global barrier 2023-04-15 02:54:41,774 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-15 02:54:41,774 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:41,775 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:41,775 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:41,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:41,775 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-15 02:54:41,775 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-15 02:54:41,775 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:41,775 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-15 02:54:41,775 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34681,1681527209133' in zk 2023-04-15 02:54:41,776 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-15 02:54:41,776 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-15 02:54:41,776 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-15 02:54:41,777 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:41,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:41,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:54:41,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:54:41,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:54:41,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:54:41,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:54:41,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:41,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:54:41,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:54:41,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:41,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34681,1681527209133': 2023-04-15 02:54:41,780 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34681,1681527209133' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-15 02:54:41,780 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-15 02:54:41,780 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-15 02:54:41,780 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-15 02:54:41,780 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-15 02:54:41,780 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-15 02:54:41,781 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:41,781 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-15 02:54:41,781 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:41,781 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:41,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:41,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:41,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:54:41,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:54:41,782 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-15 02:54:41,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-15 02:54:41,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:41,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:54:41,783 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:41,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:54:41,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:54:41,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:54:41,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:41,785 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:41,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:54:41,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:54:41,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:41,788 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-15 02:54:41,788 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:41,788 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-15 02:54:41,788 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-15 02:54:41,788 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-15 02:54:41,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-15 02:54:41,788 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-15 02:54:41,788 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-15 02:54:41,789 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-15 02:54:41,789 DEBUG [Thread-1527] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-15 02:54:41,789 DEBUG [Thread-1527] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-15 02:54:41,788 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:54:41,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-15 02:54:41,789 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:54:41,789 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:41,789 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:41,789 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:41,789 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:41,889 DEBUG [Thread-1527] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-15 02:54:41,889 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-15 02:54:41,891 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-15 02:54:41,892 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-15 02:54:41,893 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-15 02:54:41,893 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-15 02:54:41,893 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-15 02:54:41,893 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-15 02:54:41,893 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-15 02:54:41,893 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:41,893 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:54:41,894 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-15 02:54:41,894 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:41,894 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-15 02:54:41,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-15 02:54:41,894 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:41,894 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-15 02:54:41,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:54:41,894 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:41,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-15 02:54:41,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:54:41,895 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-15 02:54:41,895 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-15 02:54:41,895 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-15 02:54:41,895 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-15 02:54:41,895 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-15 02:54:41,895 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-15 02:54:41,895 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-15 02:54:41,895 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-15 02:54:41,895 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681527274083.f8160290180ba126c904acc32b60eac7. 2023-04-15 02:54:41,895 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681527274083.f8160290180ba126c904acc32b60eac7. started... 2023-04-15 02:54:41,896 INFO [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing f8160290180ba126c904acc32b60eac7 3/3 column families, dataSize=134.18 KB heapSize=380.30 KB 2023-04-15 02:54:41,896 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK f8160290180ba126c904acc32b60eac7, store=A 2023-04-15 02:54:41,896 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-15 02:54:41,896 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK f8160290180ba126c904acc32b60eac7, store=B 2023-04-15 02:54:41,896 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-15 02:54:41,896 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK f8160290180ba126c904acc32b60eac7, store=C 2023-04-15 02:54:41,896 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-15 02:54:41,982 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34681] regionserver.HRegion(9158): Flush requested on f8160290180ba126c904acc32b60eac7 2023-04-15 02:54:41,982 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681527274083.f8160290180ba126c904acc32b60eac7. as already flushing 2023-04-15 02:54:42,002 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f8160290180ba126c904acc32b60eac7, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:54:42,003 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34681] ipc.CallRunner(144): callId: 68 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41918 deadline: 1681527341998, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f8160290180ba126c904acc32b60eac7, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:42,028 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f8160290180ba126c904acc32b60eac7, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:54:42,028 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34681] ipc.CallRunner(144): callId: 58 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41940 deadline: 1681527342027, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f8160290180ba126c904acc32b60eac7, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:42,048 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f8160290180ba126c904acc32b60eac7, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:54:42,048 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34681] ipc.CallRunner(144): callId: 65 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41890 deadline: 1681527342044, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f8160290180ba126c904acc32b60eac7, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:42,052 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f8160290180ba126c904acc32b60eac7, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:54:42,053 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34681] ipc.CallRunner(144): callId: 59 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41916 deadline: 1681527342049, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f8160290180ba126c904acc32b60eac7, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:42,059 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f8160290180ba126c904acc32b60eac7, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:54:42,060 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34681] ipc.CallRunner(144): callId: 69 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41926 deadline: 1681527342057, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f8160290180ba126c904acc32b60eac7, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:42,106 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f8160290180ba126c904acc32b60eac7, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:54:42,106 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34681] ipc.CallRunner(144): callId: 70 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41918 deadline: 1681527342104, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f8160290180ba126c904acc32b60eac7, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:42,305 INFO [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=158 (bloomFilter=true), to=hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/f8160290180ba126c904acc32b60eac7/.tmp/A/b562b2c57098424c807c032a98e2362e 2023-04-15 02:54:42,311 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f8160290180ba126c904acc32b60eac7, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:54:42,311 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34681] ipc.CallRunner(144): callId: 72 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41918 deadline: 1681527342307, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f8160290180ba126c904acc32b60eac7, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:42,344 INFO [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=158 (bloomFilter=true), to=hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/f8160290180ba126c904acc32b60eac7/.tmp/B/ba80e0ee3a2249e79c5e1f0fb611e042 2023-04-15 02:54:42,535 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f8160290180ba126c904acc32b60eac7, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:54:42,535 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34681] ipc.CallRunner(144): callId: 60 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41940 deadline: 1681527342535, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f8160290180ba126c904acc32b60eac7, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:42,551 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f8160290180ba126c904acc32b60eac7, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:54:42,551 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34681] ipc.CallRunner(144): callId: 67 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41890 deadline: 1681527342550, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f8160290180ba126c904acc32b60eac7, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:42,562 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f8160290180ba126c904acc32b60eac7, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:54:42,562 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34681] ipc.CallRunner(144): callId: 61 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41916 deadline: 1681527342562, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f8160290180ba126c904acc32b60eac7, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:42,566 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f8160290180ba126c904acc32b60eac7, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:54:42,566 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34681] ipc.CallRunner(144): callId: 71 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41926 deadline: 1681527342564, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f8160290180ba126c904acc32b60eac7, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:42,616 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f8160290180ba126c904acc32b60eac7, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:54:42,616 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34681] ipc.CallRunner(144): callId: 74 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41918 deadline: 1681527342612, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f8160290180ba126c904acc32b60eac7, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:42,772 INFO [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=158 (bloomFilter=true), to=hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/f8160290180ba126c904acc32b60eac7/.tmp/C/3f1b6a9966784dbc8e885ce9ea199958 2023-04-15 02:54:42,776 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/f8160290180ba126c904acc32b60eac7/.tmp/A/b562b2c57098424c807c032a98e2362e as hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/f8160290180ba126c904acc32b60eac7/A/b562b2c57098424c807c032a98e2362e 2023-04-15 02:54:42,781 INFO [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/f8160290180ba126c904acc32b60eac7/A/b562b2c57098424c807c032a98e2362e, entries=150, sequenceid=158, filesize=11.8 K 2023-04-15 02:54:42,782 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/f8160290180ba126c904acc32b60eac7/.tmp/B/ba80e0ee3a2249e79c5e1f0fb611e042 as hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/f8160290180ba126c904acc32b60eac7/B/ba80e0ee3a2249e79c5e1f0fb611e042 2023-04-15 02:54:42,787 INFO [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/f8160290180ba126c904acc32b60eac7/B/ba80e0ee3a2249e79c5e1f0fb611e042, entries=150, sequenceid=158, filesize=11.8 K 2023-04-15 02:54:42,788 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/f8160290180ba126c904acc32b60eac7/.tmp/C/3f1b6a9966784dbc8e885ce9ea199958 as hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/f8160290180ba126c904acc32b60eac7/C/3f1b6a9966784dbc8e885ce9ea199958 2023-04-15 02:54:42,792 INFO [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/f8160290180ba126c904acc32b60eac7/C/3f1b6a9966784dbc8e885ce9ea199958, entries=150, sequenceid=158, filesize=11.8 K 2023-04-15 02:54:42,793 INFO [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~134.18 KB/137400, heapSize ~380.25 KB/389376, currentSize=46.96 KB/48090 for f8160290180ba126c904acc32b60eac7 in 897ms, sequenceid=158, compaction requested=true 2023-04-15 02:54:42,793 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for f8160290180ba126c904acc32b60eac7: 2023-04-15 02:54:42,793 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681527274083.f8160290180ba126c904acc32b60eac7. 2023-04-15 02:54:42,793 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-15 02:54:42,793 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-15 02:54:42,793 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:42,794 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-15 02:54:42,794 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34681,1681527209133' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-15 02:54:42,797 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:42,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:42,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:54:42,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:54:42,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:54:42,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:54:42,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:54:42,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:42,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:54:42,799 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34681,1681527209133' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-15 02:54:42,799 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@171babff[Count = 0] remaining members to acquire global barrier 2023-04-15 02:54:42,799 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-15 02:54:42,799 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:42,803 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:42,803 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:42,803 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] zookeeper.ZKUtil(162): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:42,804 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:42,804 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-15 02:54:42,804 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-15 02:54:42,804 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-15 02:54:42,804 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34681,1681527209133' in zk 2023-04-15 02:54:42,803 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-15 02:54:42,805 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:42,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:42,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:54:42,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:54:42,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:54:42,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:54:42,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:54:42,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:42,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:54:42,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:54:42,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:42,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34681,1681527209133': 2023-04-15 02:54:42,809 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34681,1681527209133' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-15 02:54:42,809 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-15 02:54:42,809 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-15 02:54:42,809 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-15 02:54:42,809 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-15 02:54:42,809 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-15 02:54:42,810 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-15 02:54:42,810 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-15 02:54:42,810 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-15 02:54:42,810 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:42,810 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-15 02:54:42,810 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:42,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:42,810 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:42,811 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-15 02:54:42,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-15 02:54:42,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:42,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:54:42,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:54:42,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:42,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:54:42,812 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:42,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:54:42,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:54:42,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:54:42,814 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:42,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:42,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:54:42,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:54:42,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:42,816 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:42,816 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-15 02:54:42,816 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-15 02:54:42,816 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-15 02:54:42,816 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-15 02:54:42,816 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:54:42,817 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-15 02:54:42,817 DEBUG [Thread-1527] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-15 02:54:42,816 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-15 02:54:42,816 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:42,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-15 02:54:42,817 DEBUG [Thread-1527] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-15 02:54:42,817 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:54:42,817 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:42,817 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:42,817 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:42,817 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:42,817 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-15 02:54:42,817 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-15 02:54:42,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-15 02:54:42,917 DEBUG [Thread-1527] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-15 02:54:42,918 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-15 02:54:42,919 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-15 02:54:42,921 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-15 02:54:42,921 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-15 02:54:42,921 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-15 02:54:42,921 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-15 02:54:42,921 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-15 02:54:42,921 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-15 02:54:42,921 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:42,921 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:54:42,922 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-15 02:54:42,922 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:42,922 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-15 02:54:42,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-15 02:54:42,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:54:42,923 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:42,923 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-15 02:54:42,923 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:42,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-15 02:54:42,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:54:42,923 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-15 02:54:42,924 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-15 02:54:42,924 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-15 02:54:42,924 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-15 02:54:42,924 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-15 02:54:42,924 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-15 02:54:42,924 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-15 02:54:42,924 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-15 02:54:42,924 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681527274083.f8160290180ba126c904acc32b60eac7. 2023-04-15 02:54:42,924 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681527274083.f8160290180ba126c904acc32b60eac7. started... 2023-04-15 02:54:42,924 INFO [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing f8160290180ba126c904acc32b60eac7 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-15 02:54:42,925 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK f8160290180ba126c904acc32b60eac7, store=A 2023-04-15 02:54:42,925 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-15 02:54:42,925 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK f8160290180ba126c904acc32b60eac7, store=B 2023-04-15 02:54:42,925 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-15 02:54:42,925 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK f8160290180ba126c904acc32b60eac7, store=C 2023-04-15 02:54:42,925 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-15 02:54:42,941 INFO [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=168 (bloomFilter=true), to=hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/f8160290180ba126c904acc32b60eac7/.tmp/A/f9cf581ca3df4acdad0e1c123fb09953 2023-04-15 02:54:42,972 INFO [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=168 (bloomFilter=true), to=hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/f8160290180ba126c904acc32b60eac7/.tmp/B/1e7004cf2a474345befc018fc8adcf7f 2023-04-15 02:54:42,996 INFO [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=168 (bloomFilter=true), to=hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/f8160290180ba126c904acc32b60eac7/.tmp/C/11913411dbc84ae3a331aaf69d76e2ea 2023-04-15 02:54:43,003 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/f8160290180ba126c904acc32b60eac7/.tmp/A/f9cf581ca3df4acdad0e1c123fb09953 as hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/f8160290180ba126c904acc32b60eac7/A/f9cf581ca3df4acdad0e1c123fb09953 2023-04-15 02:54:43,007 INFO [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/f8160290180ba126c904acc32b60eac7/A/f9cf581ca3df4acdad0e1c123fb09953, entries=150, sequenceid=168, filesize=11.8 K 2023-04-15 02:54:43,009 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/f8160290180ba126c904acc32b60eac7/.tmp/B/1e7004cf2a474345befc018fc8adcf7f as hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/f8160290180ba126c904acc32b60eac7/B/1e7004cf2a474345befc018fc8adcf7f 2023-04-15 02:54:43,027 INFO [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/f8160290180ba126c904acc32b60eac7/B/1e7004cf2a474345befc018fc8adcf7f, entries=150, sequenceid=168, filesize=11.8 K 2023-04-15 02:54:43,028 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/f8160290180ba126c904acc32b60eac7/.tmp/C/11913411dbc84ae3a331aaf69d76e2ea as hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/f8160290180ba126c904acc32b60eac7/C/11913411dbc84ae3a331aaf69d76e2ea 2023-04-15 02:54:43,031 INFO [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/f8160290180ba126c904acc32b60eac7/C/11913411dbc84ae3a331aaf69d76e2ea, entries=150, sequenceid=168, filesize=11.8 K 2023-04-15 02:54:43,032 INFO [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=0 B/0 for f8160290180ba126c904acc32b60eac7 in 108ms, sequenceid=168, compaction requested=true 2023-04-15 02:54:43,032 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for f8160290180ba126c904acc32b60eac7: 2023-04-15 02:54:43,032 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681527274083.f8160290180ba126c904acc32b60eac7. 2023-04-15 02:54:43,032 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-15 02:54:43,032 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-15 02:54:43,032 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:43,032 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-15 02:54:43,032 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34681,1681527209133' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-15 02:54:43,033 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:43,033 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:43,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:43,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:54:43,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:54:43,033 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:43,033 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-15 02:54:43,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:54:43,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:54:43,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:54:43,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:43,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:54:43,035 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34681,1681527209133' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-15 02:54:43,035 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3d4859a5[Count = 0] remaining members to acquire global barrier 2023-04-15 02:54:43,035 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-15 02:54:43,035 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:43,036 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:43,036 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:43,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:43,036 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-15 02:54:43,036 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-15 02:54:43,036 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34681,1681527209133' in zk 2023-04-15 02:54:43,036 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:43,036 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-15 02:54:43,037 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-15 02:54:43,037 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-15 02:54:43,037 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:43,037 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-15 02:54:43,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:43,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:54:43,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:54:43,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:54:43,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:54:43,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:54:43,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:43,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:54:43,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:54:43,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:43,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34681,1681527209133': 2023-04-15 02:54:43,040 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34681,1681527209133' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-15 02:54:43,040 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-15 02:54:43,040 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-15 02:54:43,040 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-15 02:54:43,040 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-15 02:54:43,040 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-15 02:54:43,041 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:43,041 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-15 02:54:43,041 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:43,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:43,041 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:43,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:43,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:54:43,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:54:43,041 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-15 02:54:43,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-15 02:54:43,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:54:43,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:43,042 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:43,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:54:43,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:54:43,043 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:43,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:54:43,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:43,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:54:43,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:54:43,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:43,045 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:43,045 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:54:43,045 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:54:43,045 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-15 02:54:43,045 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-15 02:54:43,045 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-15 02:54:43,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-15 02:54:43,046 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-15 02:54:43,046 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:43,046 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-15 02:54:43,046 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-15 02:54:43,046 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:43,046 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-15 02:54:43,046 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:43,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-15 02:54:43,046 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:43,046 DEBUG [Thread-1527] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-15 02:54:43,046 DEBUG [Thread-1527] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-15 02:54:43,046 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-15 02:54:43,138 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34681] regionserver.HRegion(9158): Flush requested on f8160290180ba126c904acc32b60eac7 2023-04-15 02:54:43,138 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing f8160290180ba126c904acc32b60eac7 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-15 02:54:43,138 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK f8160290180ba126c904acc32b60eac7, store=A 2023-04-15 02:54:43,139 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-15 02:54:43,139 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK f8160290180ba126c904acc32b60eac7, store=B 2023-04-15 02:54:43,139 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-15 02:54:43,139 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK f8160290180ba126c904acc32b60eac7, store=C 2023-04-15 02:54:43,139 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-15 02:54:43,146 DEBUG [Thread-1527] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-15 02:54:43,147 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-15 02:54:43,152 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-15 02:54:43,158 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-15 02:54:43,158 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-15 02:54:43,158 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-15 02:54:43,158 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-15 02:54:43,158 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-15 02:54:43,158 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-15 02:54:43,159 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:43,159 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:54:43,162 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-15 02:54:43,162 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:43,162 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-15 02:54:43,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-15 02:54:43,163 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:43,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:54:43,163 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-15 02:54:43,163 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:43,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-15 02:54:43,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:54:43,164 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-15 02:54:43,164 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-15 02:54:43,164 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-15 02:54:43,164 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-15 02:54:43,164 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-15 02:54:43,164 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-15 02:54:43,166 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-15 02:54:43,166 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681527274083.f8160290180ba126c904acc32b60eac7. 2023-04-15 02:54:43,166 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681527274083.f8160290180ba126c904acc32b60eac7. started... 2023-04-15 02:54:43,166 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-15 02:54:43,166 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681527274083.f8160290180ba126c904acc32b60eac7. as already flushing 2023-04-15 02:54:43,166 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681527274083.f8160290180ba126c904acc32b60eac7. 2023-04-15 02:54:43,166 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=178 (bloomFilter=true), to=hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/f8160290180ba126c904acc32b60eac7/.tmp/A/7d6f907648d94f078af742e9bed125b2 2023-04-15 02:54:43,166 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-15 02:54:43,166 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-15 02:54:43,166 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:43,166 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-15 02:54:43,166 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34681,1681527209133' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-15 02:54:43,167 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:43,167 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:43,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:43,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:54:43,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:54:43,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:54:43,168 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:43,168 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-15 02:54:43,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:54:43,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:54:43,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:43,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:54:43,171 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34681,1681527209133' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-15 02:54:43,171 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@79ba9b62[Count = 0] remaining members to acquire global barrier 2023-04-15 02:54:43,171 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-15 02:54:43,171 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:43,171 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:43,171 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:43,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:43,172 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-15 02:54:43,172 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-15 02:54:43,172 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34681,1681527209133' in zk 2023-04-15 02:54:43,172 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:43,172 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-15 02:54:43,173 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:43,173 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-15 02:54:43,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:43,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:54:43,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:54:43,173 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-15 02:54:43,173 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-15 02:54:43,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:54:43,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:54:43,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:54:43,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:43,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:54:43,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:54:43,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:43,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34681,1681527209133': 2023-04-15 02:54:43,175 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34681,1681527209133' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-15 02:54:43,175 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-15 02:54:43,175 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-15 02:54:43,175 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-15 02:54:43,175 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-15 02:54:43,175 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-15 02:54:43,176 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:43,176 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:43,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:43,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:54:43,176 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:43,176 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-15 02:54:43,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:43,177 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-15 02:54:43,177 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:43,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:54:43,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-15 02:54:43,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:43,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:54:43,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:54:43,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:54:43,178 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:43,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:54:43,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:43,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:54:43,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:54:43,180 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-15 02:54:43,181 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-15 02:54:43,181 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-15 02:54:43,181 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:43,181 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:54:43,181 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:54:43,181 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:43,181 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:43,181 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:43,181 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:43,181 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-15 02:54:43,181 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-15 02:54:43,181 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-15 02:54:43,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-15 02:54:43,181 DEBUG [Thread-1527] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-15 02:54:43,181 DEBUG [Thread-1527] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-15 02:54:43,181 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-15 02:54:43,181 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-15 02:54:43,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-15 02:54:43,209 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681527283148, max timestamp=1681527283207Num uniques 859; 2023-04-15 02:54:43,209 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-5] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-15 02:54:43,209 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681527283148, max timestamp=1681527283207Num uniques 859; 2023-04-15 02:54:43,209 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681527283148, max timestamp=1681527283207Num uniques 859; 2023-04-15 02:54:43,209 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-4] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-15 02:54:43,209 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-1] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-15 02:54:43,281 DEBUG [Thread-1527] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-15 02:54:43,282 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-15 02:54:43,283 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-15 02:54:43,290 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-15 02:54:43,290 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-15 02:54:43,290 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-15 02:54:43,290 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-15 02:54:43,290 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-15 02:54:43,290 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-15 02:54:43,290 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:43,290 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:54:43,291 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f8160290180ba126c904acc32b60eac7, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:54:43,291 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34681] ipc.CallRunner(144): callId: 117 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41918 deadline: 1681527343287, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f8160290180ba126c904acc32b60eac7, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:43,292 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-15 02:54:43,292 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-15 02:54:43,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-15 02:54:43,292 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:43,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:54:43,292 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:43,292 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-15 02:54:43,293 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:43,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-15 02:54:43,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:54:43,293 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-15 02:54:43,294 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-15 02:54:43,294 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-15 02:54:43,295 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-15 02:54:43,296 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-15 02:54:43,296 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-15 02:54:43,297 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-15 02:54:43,297 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-15 02:54:43,297 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681527274083.f8160290180ba126c904acc32b60eac7. 2023-04-15 02:54:43,297 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681527274083.f8160290180ba126c904acc32b60eac7. started... 2023-04-15 02:54:43,297 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681527274083.f8160290180ba126c904acc32b60eac7. as already flushing 2023-04-15 02:54:43,297 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681527274083.f8160290180ba126c904acc32b60eac7. 2023-04-15 02:54:43,297 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-15 02:54:43,297 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-15 02:54:43,297 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:43,297 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-15 02:54:43,297 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34681,1681527209133' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-15 02:54:43,299 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:43,299 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:43,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:43,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:54:43,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:54:43,300 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:43,300 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-15 02:54:43,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:54:43,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:54:43,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:54:43,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:43,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:54:43,301 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34681,1681527209133' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-15 02:54:43,301 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4d7c511[Count = 0] remaining members to acquire global barrier 2023-04-15 02:54:43,301 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-15 02:54:43,301 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:43,301 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:43,302 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:43,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:43,302 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-15 02:54:43,302 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-15 02:54:43,302 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34681,1681527209133' in zk 2023-04-15 02:54:43,302 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:43,302 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-15 02:54:43,302 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:43,302 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-15 02:54:43,303 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-15 02:54:43,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:43,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:54:43,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:54:43,303 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-15 02:54:43,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:54:43,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:54:43,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:54:43,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:43,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:54:43,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:54:43,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:43,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34681,1681527209133': 2023-04-15 02:54:43,305 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34681,1681527209133' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-15 02:54:43,305 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-15 02:54:43,305 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-15 02:54:43,305 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-15 02:54:43,305 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-15 02:54:43,305 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-15 02:54:43,306 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:43,306 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:43,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:43,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:54:43,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:54:43,306 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:43,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:43,306 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-15 02:54:43,307 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-15 02:54:43,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-15 02:54:43,307 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:43,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:43,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:54:43,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:54:43,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:54:43,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:54:43,309 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:43,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:43,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:54:43,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:54:43,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:43,310 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-15 02:54:43,311 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-15 02:54:43,311 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-15 02:54:43,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-15 02:54:43,311 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:43,311 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:54:43,311 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:54:43,311 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-15 02:54:43,311 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-15 02:54:43,311 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-15 02:54:43,311 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-15 02:54:43,311 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-15 02:54:43,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-15 02:54:43,311 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:43,311 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:43,311 DEBUG [Thread-1527] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-15 02:54:43,311 DEBUG [Thread-1527] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-15 02:54:43,311 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:43,311 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:43,397 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f8160290180ba126c904acc32b60eac7, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:54:43,397 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34681] ipc.CallRunner(144): callId: 119 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41918 deadline: 1681527343392, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f8160290180ba126c904acc32b60eac7, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:43,411 DEBUG [Thread-1527] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-15 02:54:43,412 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-15 02:54:43,413 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-15 02:54:43,418 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-15 02:54:43,418 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-15 02:54:43,418 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-15 02:54:43,418 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-15 02:54:43,418 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-15 02:54:43,418 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-15 02:54:43,419 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:43,419 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:54:43,419 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:43,419 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-15 02:54:43,419 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-15 02:54:43,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-15 02:54:43,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:54:43,420 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:43,420 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-15 02:54:43,420 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:43,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-15 02:54:43,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:54:43,420 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-15 02:54:43,421 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-15 02:54:43,421 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-15 02:54:43,421 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-15 02:54:43,421 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-15 02:54:43,421 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-15 02:54:43,426 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-15 02:54:43,426 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-15 02:54:43,426 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681527274083.f8160290180ba126c904acc32b60eac7. 2023-04-15 02:54:43,426 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681527274083.f8160290180ba126c904acc32b60eac7. started... 2023-04-15 02:54:43,426 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681527274083.f8160290180ba126c904acc32b60eac7. as already flushing 2023-04-15 02:54:43,426 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681527274083.f8160290180ba126c904acc32b60eac7. 2023-04-15 02:54:43,426 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-15 02:54:43,426 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-15 02:54:43,426 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:43,426 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-15 02:54:43,426 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34681,1681527209133' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-15 02:54:43,433 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:43,433 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:43,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:43,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:54:43,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:54:43,434 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:43,434 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-15 02:54:43,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:54:43,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:54:43,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:54:43,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:43,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:54:43,435 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34681,1681527209133' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-15 02:54:43,435 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4ed9d227[Count = 0] remaining members to acquire global barrier 2023-04-15 02:54:43,435 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-15 02:54:43,435 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:43,437 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:43,437 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-15 02:54:43,438 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:43,438 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:43,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:43,438 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-15 02:54:43,438 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-15 02:54:43,438 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34681,1681527209133' in zk 2023-04-15 02:54:43,439 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-15 02:54:43,439 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-15 02:54:43,439 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-15 02:54:43,439 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:43,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:43,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:54:43,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:54:43,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:54:43,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:54:43,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:54:43,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:43,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:54:43,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:54:43,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:43,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34681,1681527209133': 2023-04-15 02:54:43,441 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34681,1681527209133' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-15 02:54:43,441 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-15 02:54:43,441 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-15 02:54:43,441 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-15 02:54:43,441 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-15 02:54:43,441 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-15 02:54:43,443 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:43,443 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-15 02:54:43,443 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:43,443 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:43,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:43,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:43,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:54:43,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:54:43,444 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-15 02:54:43,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-15 02:54:43,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:54:43,444 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:43,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:43,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:54:43,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:54:43,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:54:43,445 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:43,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:43,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:54:43,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:54:43,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:43,446 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:43,446 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:54:43,446 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:54:43,446 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-15 02:54:43,446 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-15 02:54:43,446 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-15 02:54:43,446 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:43,446 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:43,446 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:43,447 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:43,446 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-15 02:54:43,447 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-15 02:54:43,447 DEBUG [Thread-1527] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-15 02:54:43,447 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-15 02:54:43,447 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-15 02:54:43,447 DEBUG [Thread-1527] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-15 02:54:43,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-15 02:54:43,447 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-15 02:54:43,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-15 02:54:43,547 DEBUG [Thread-1527] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-15 02:54:43,547 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-15 02:54:43,548 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f8160290180ba126c904acc32b60eac7, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:54:43,548 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34681] ipc.CallRunner(144): callId: 62 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41940 deadline: 1681527343545, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f8160290180ba126c904acc32b60eac7, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:43,549 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-15 02:54:43,550 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-15 02:54:43,550 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-15 02:54:43,550 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-15 02:54:43,550 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-15 02:54:43,550 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-15 02:54:43,550 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-15 02:54:43,550 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:43,550 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:54:43,551 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-15 02:54:43,551 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:43,551 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-15 02:54:43,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-15 02:54:43,551 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:43,552 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-15 02:54:43,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:54:43,553 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:43,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-15 02:54:43,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:54:43,553 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-15 02:54:43,553 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-15 02:54:43,553 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-15 02:54:43,553 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-15 02:54:43,553 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-15 02:54:43,553 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-15 02:54:43,554 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-15 02:54:43,554 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-15 02:54:43,554 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681527274083.f8160290180ba126c904acc32b60eac7. 2023-04-15 02:54:43,554 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681527274083.f8160290180ba126c904acc32b60eac7. started... 2023-04-15 02:54:43,554 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681527274083.f8160290180ba126c904acc32b60eac7. as already flushing 2023-04-15 02:54:43,554 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681527274083.f8160290180ba126c904acc32b60eac7. 2023-04-15 02:54:43,554 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-15 02:54:43,554 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-15 02:54:43,554 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:43,554 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-15 02:54:43,554 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34681,1681527209133' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-15 02:54:43,556 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f8160290180ba126c904acc32b60eac7, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:54:43,556 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34681] ipc.CallRunner(144): callId: 69 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41890 deadline: 1681527343554, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f8160290180ba126c904acc32b60eac7, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:43,570 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:43,570 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:43,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:43,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:54:43,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:54:43,570 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:43,570 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-15 02:54:43,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:54:43,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:54:43,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:54:43,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:43,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:54:43,572 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34681,1681527209133' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-15 02:54:43,572 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@31232f28[Count = 0] remaining members to acquire global barrier 2023-04-15 02:54:43,572 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-15 02:54:43,572 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:43,573 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:43,573 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:43,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:43,573 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-15 02:54:43,573 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-15 02:54:43,573 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:43,573 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-15 02:54:43,573 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34681,1681527209133' in zk 2023-04-15 02:54:43,573 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f8160290180ba126c904acc32b60eac7, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:54:43,574 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34681] ipc.CallRunner(144): callId: 63 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41916 deadline: 1681527343572, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f8160290180ba126c904acc32b60eac7, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:43,574 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-15 02:54:43,574 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-15 02:54:43,574 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:43,574 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-15 02:54:43,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:43,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:54:43,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:54:43,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:54:43,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:54:43,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:54:43,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:43,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:54:43,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:54:43,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:43,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34681,1681527209133': 2023-04-15 02:54:43,577 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34681,1681527209133' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-15 02:54:43,577 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-15 02:54:43,577 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-15 02:54:43,577 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-15 02:54:43,577 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-15 02:54:43,577 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-15 02:54:43,578 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f8160290180ba126c904acc32b60eac7, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:54:43,578 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34681] ipc.CallRunner(144): callId: 73 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41926 deadline: 1681527343576, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f8160290180ba126c904acc32b60eac7, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:43,579 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:43,579 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:43,579 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:43,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:43,579 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-15 02:54:43,579 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-15 02:54:43,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-15 02:54:43,579 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:43,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:43,580 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:43,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:43,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:54:43,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:54:43,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:54:43,583 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-15 02:54:43,583 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-15 02:54:43,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:54:43,583 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:43,583 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-15 02:54:43,583 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-15 02:54:43,583 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-15 02:54:43,583 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-15 02:54:43,583 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:54:43,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-15 02:54:43,583 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:54:43,583 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:43,583 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:43,583 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:43,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:54:43,583 DEBUG [Thread-1527] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-15 02:54:43,583 DEBUG [Thread-1527] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-15 02:54:43,583 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-15 02:54:43,583 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:43,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-15 02:54:43,584 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-15 02:54:43,588 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=178 (bloomFilter=true), to=hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/f8160290180ba126c904acc32b60eac7/.tmp/B/2b8f9e91423a47e59fbc1a268696347b 2023-04-15 02:54:43,599 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=178 (bloomFilter=true), to=hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/f8160290180ba126c904acc32b60eac7/.tmp/C/6ca7e5c7d6c742f49e993eb5d01c790c 2023-04-15 02:54:43,602 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f8160290180ba126c904acc32b60eac7, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:54:43,602 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34681] ipc.CallRunner(144): callId: 121 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41918 deadline: 1681527343599, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f8160290180ba126c904acc32b60eac7, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:43,604 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/f8160290180ba126c904acc32b60eac7/.tmp/A/7d6f907648d94f078af742e9bed125b2 as hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/f8160290180ba126c904acc32b60eac7/A/7d6f907648d94f078af742e9bed125b2 2023-04-15 02:54:43,608 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/f8160290180ba126c904acc32b60eac7/A/7d6f907648d94f078af742e9bed125b2, entries=200, sequenceid=178, filesize=14.1 K 2023-04-15 02:54:43,609 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/f8160290180ba126c904acc32b60eac7/.tmp/B/2b8f9e91423a47e59fbc1a268696347b as hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/f8160290180ba126c904acc32b60eac7/B/2b8f9e91423a47e59fbc1a268696347b 2023-04-15 02:54:43,616 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/f8160290180ba126c904acc32b60eac7/B/2b8f9e91423a47e59fbc1a268696347b, entries=150, sequenceid=178, filesize=11.8 K 2023-04-15 02:54:43,616 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/f8160290180ba126c904acc32b60eac7/.tmp/C/6ca7e5c7d6c742f49e993eb5d01c790c as hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/f8160290180ba126c904acc32b60eac7/C/6ca7e5c7d6c742f49e993eb5d01c790c 2023-04-15 02:54:43,623 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/f8160290180ba126c904acc32b60eac7/C/6ca7e5c7d6c742f49e993eb5d01c790c, entries=150, sequenceid=178, filesize=11.8 K 2023-04-15 02:54:43,623 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=134.18 KB/137400 for f8160290180ba126c904acc32b60eac7 in 485ms, sequenceid=178, compaction requested=true 2023-04-15 02:54:43,624 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for f8160290180ba126c904acc32b60eac7: 2023-04-15 02:54:43,624 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-15 02:54:43,624 DEBUG [RS:0;jenkins-hbase20:34681-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 6 store files, 0 compacting, 6 eligible, 16 blocking 2023-04-15 02:54:43,624 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-15 02:54:43,624 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=2), splitQueue=0 2023-04-15 02:54:43,624 DEBUG [RS:0;jenkins-hbase20:34681-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 6 store files, 0 compacting, 6 eligible, 16 blocking 2023-04-15 02:54:43,634 DEBUG [RS:0;jenkins-hbase20:34681-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 6 files of size 71904 starting at candidate #0 after considering 10 permutations with 10 in ratio 2023-04-15 02:54:43,634 DEBUG [RS:0;jenkins-hbase20:34681-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 6 files of size 74294 starting at candidate #0 after considering 10 permutations with 10 in ratio 2023-04-15 02:54:43,634 DEBUG [RS:0;jenkins-hbase20:34681-longCompactions-0] regionserver.HStore(1912): f8160290180ba126c904acc32b60eac7/B is initiating minor compaction (all files) 2023-04-15 02:54:43,634 DEBUG [RS:0;jenkins-hbase20:34681-shortCompactions-0] regionserver.HStore(1912): f8160290180ba126c904acc32b60eac7/A is initiating minor compaction (all files) 2023-04-15 02:54:43,634 INFO [RS:0;jenkins-hbase20:34681-longCompactions-0] regionserver.HRegion(2259): Starting compaction of f8160290180ba126c904acc32b60eac7/B in TestAcidGuarantees,,1681527274083.f8160290180ba126c904acc32b60eac7. 2023-04-15 02:54:43,634 INFO [RS:0;jenkins-hbase20:34681-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of f8160290180ba126c904acc32b60eac7/A in TestAcidGuarantees,,1681527274083.f8160290180ba126c904acc32b60eac7. 2023-04-15 02:54:43,634 INFO [RS:0;jenkins-hbase20:34681-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/f8160290180ba126c904acc32b60eac7/A/61dcb71e9705420ab52f98869f5eab5a, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/f8160290180ba126c904acc32b60eac7/A/0944ea0d48e048fe8486e8aa032e1470, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/f8160290180ba126c904acc32b60eac7/A/3252d9d39f894e3e98cd1a5aa3e3014d, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/f8160290180ba126c904acc32b60eac7/A/b562b2c57098424c807c032a98e2362e, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/f8160290180ba126c904acc32b60eac7/A/f9cf581ca3df4acdad0e1c123fb09953, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/f8160290180ba126c904acc32b60eac7/A/7d6f907648d94f078af742e9bed125b2] into tmpdir=hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/f8160290180ba126c904acc32b60eac7/.tmp, totalSize=72.6 K 2023-04-15 02:54:43,634 INFO [RS:0;jenkins-hbase20:34681-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/f8160290180ba126c904acc32b60eac7/B/2e47e14a986e4838b5afda846a0524da, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/f8160290180ba126c904acc32b60eac7/B/0b2852aa58d9444994a5a57c93b0d068, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/f8160290180ba126c904acc32b60eac7/B/b279733ba1f64aa5af8bfe1a1e7e0cb5, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/f8160290180ba126c904acc32b60eac7/B/ba80e0ee3a2249e79c5e1f0fb611e042, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/f8160290180ba126c904acc32b60eac7/B/1e7004cf2a474345befc018fc8adcf7f, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/f8160290180ba126c904acc32b60eac7/B/2b8f9e91423a47e59fbc1a268696347b] into tmpdir=hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/f8160290180ba126c904acc32b60eac7/.tmp, totalSize=70.2 K 2023-04-15 02:54:43,634 DEBUG [RS:0;jenkins-hbase20:34681-shortCompactions-0] compactions.Compactor(207): Compacting 61dcb71e9705420ab52f98869f5eab5a, keycount=150, bloomtype=ROW, size=11.7 K, encoding=NONE, compression=NONE, seqNum=73, earliestPutTs=1681527279139 2023-04-15 02:54:43,635 DEBUG [RS:0;jenkins-hbase20:34681-longCompactions-0] compactions.Compactor(207): Compacting 2e47e14a986e4838b5afda846a0524da, keycount=150, bloomtype=ROW, size=11.7 K, encoding=NONE, compression=NONE, seqNum=73, earliestPutTs=1681527279139 2023-04-15 02:54:43,635 DEBUG [RS:0;jenkins-hbase20:34681-longCompactions-0] compactions.Compactor(207): Compacting 0b2852aa58d9444994a5a57c93b0d068, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=110, earliestPutTs=1681527279180 2023-04-15 02:54:43,635 DEBUG [RS:0;jenkins-hbase20:34681-shortCompactions-0] compactions.Compactor(207): Compacting 0944ea0d48e048fe8486e8aa032e1470, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=110, earliestPutTs=1681527279180 2023-04-15 02:54:43,635 DEBUG [RS:0;jenkins-hbase20:34681-longCompactions-0] compactions.Compactor(207): Compacting b279733ba1f64aa5af8bfe1a1e7e0cb5, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=122, earliestPutTs=1681527280288 2023-04-15 02:54:43,635 DEBUG [RS:0;jenkins-hbase20:34681-shortCompactions-0] compactions.Compactor(207): Compacting 3252d9d39f894e3e98cd1a5aa3e3014d, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=122, earliestPutTs=1681527280288 2023-04-15 02:54:43,636 DEBUG [RS:0;jenkins-hbase20:34681-longCompactions-0] compactions.Compactor(207): Compacting ba80e0ee3a2249e79c5e1f0fb611e042, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=158, earliestPutTs=1681527281416 2023-04-15 02:54:43,636 DEBUG [RS:0;jenkins-hbase20:34681-shortCompactions-0] compactions.Compactor(207): Compacting b562b2c57098424c807c032a98e2362e, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=158, earliestPutTs=1681527281416 2023-04-15 02:54:43,636 DEBUG [RS:0;jenkins-hbase20:34681-longCompactions-0] compactions.Compactor(207): Compacting 1e7004cf2a474345befc018fc8adcf7f, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=168, earliestPutTs=1681527281981 2023-04-15 02:54:43,636 DEBUG [RS:0;jenkins-hbase20:34681-shortCompactions-0] compactions.Compactor(207): Compacting f9cf581ca3df4acdad0e1c123fb09953, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=168, earliestPutTs=1681527281981 2023-04-15 02:54:43,636 DEBUG [RS:0;jenkins-hbase20:34681-longCompactions-0] compactions.Compactor(207): Compacting 2b8f9e91423a47e59fbc1a268696347b, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=178, earliestPutTs=1681527283131 2023-04-15 02:54:43,637 DEBUG [RS:0;jenkins-hbase20:34681-shortCompactions-0] compactions.Compactor(207): Compacting 7d6f907648d94f078af742e9bed125b2, keycount=200, bloomtype=ROW, size=14.1 K, encoding=NONE, compression=NONE, seqNum=178, earliestPutTs=1681527283123 2023-04-15 02:54:43,660 INFO [RS:0;jenkins-hbase20:34681-shortCompactions-0] throttle.PressureAwareThroughputController(145): f8160290180ba126c904acc32b60eac7#A#compaction#209 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-15 02:54:43,668 INFO [RS:0;jenkins-hbase20:34681-longCompactions-0] throttle.PressureAwareThroughputController(145): f8160290180ba126c904acc32b60eac7#B#compaction#210 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-15 02:54:43,683 DEBUG [Thread-1527] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-15 02:54:43,684 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-15 02:54:43,686 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-15 02:54:43,694 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-15 02:54:43,694 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-15 02:54:43,694 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-15 02:54:43,694 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-15 02:54:43,694 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-15 02:54:43,694 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-15 02:54:43,694 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:43,695 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:54:43,695 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-15 02:54:43,695 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:43,695 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-15 02:54:43,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-15 02:54:43,696 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:43,696 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-15 02:54:43,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:54:43,696 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:43,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-15 02:54:43,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:54:43,696 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-15 02:54:43,714 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-15 02:54:43,714 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-15 02:54:43,714 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-15 02:54:43,714 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-15 02:54:43,714 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-15 02:54:43,718 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-15 02:54:43,718 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-15 02:54:43,718 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681527274083.f8160290180ba126c904acc32b60eac7. 2023-04-15 02:54:43,718 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681527274083.f8160290180ba126c904acc32b60eac7. started... 2023-04-15 02:54:43,718 INFO [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing f8160290180ba126c904acc32b60eac7 3/3 column families, dataSize=134.18 KB heapSize=384.98 KB 2023-04-15 02:54:43,734 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK f8160290180ba126c904acc32b60eac7, store=A 2023-04-15 02:54:43,734 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-15 02:54:43,734 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK f8160290180ba126c904acc32b60eac7, store=B 2023-04-15 02:54:43,734 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-15 02:54:43,734 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK f8160290180ba126c904acc32b60eac7, store=C 2023-04-15 02:54:43,734 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-15 02:54:43,742 DEBUG [RS:0;jenkins-hbase20:34681-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/f8160290180ba126c904acc32b60eac7/.tmp/B/fffc50bcd93a4e7e9a049c6eddc9b678 as hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/f8160290180ba126c904acc32b60eac7/B/fffc50bcd93a4e7e9a049c6eddc9b678 2023-04-15 02:54:43,750 INFO [RS:0;jenkins-hbase20:34681-longCompactions-0] regionserver.HStore(1652): Completed compaction of 6 (all) file(s) in f8160290180ba126c904acc32b60eac7/B of f8160290180ba126c904acc32b60eac7 into fffc50bcd93a4e7e9a049c6eddc9b678(size=12.1 K), total size for store is 12.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-15 02:54:43,751 DEBUG [RS:0;jenkins-hbase20:34681-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for f8160290180ba126c904acc32b60eac7: 2023-04-15 02:54:43,751 INFO [RS:0;jenkins-hbase20:34681-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681527274083.f8160290180ba126c904acc32b60eac7., storeName=f8160290180ba126c904acc32b60eac7/B, priority=10, startTime=1681527283624; duration=0sec 2023-04-15 02:54:43,751 DEBUG [RS:0;jenkins-hbase20:34681-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-15 02:54:43,751 DEBUG [RS:0;jenkins-hbase20:34681-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 6 store files, 0 compacting, 6 eligible, 16 blocking 2023-04-15 02:54:43,756 DEBUG [RS:0;jenkins-hbase20:34681-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 6 files of size 71904 starting at candidate #0 after considering 10 permutations with 10 in ratio 2023-04-15 02:54:43,757 DEBUG [RS:0;jenkins-hbase20:34681-longCompactions-0] regionserver.HStore(1912): f8160290180ba126c904acc32b60eac7/C is initiating minor compaction (all files) 2023-04-15 02:54:43,757 INFO [RS:0;jenkins-hbase20:34681-longCompactions-0] regionserver.HRegion(2259): Starting compaction of f8160290180ba126c904acc32b60eac7/C in TestAcidGuarantees,,1681527274083.f8160290180ba126c904acc32b60eac7. 2023-04-15 02:54:43,757 INFO [RS:0;jenkins-hbase20:34681-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/f8160290180ba126c904acc32b60eac7/C/29656fb2fd6c4e998f291542df8d67fc, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/f8160290180ba126c904acc32b60eac7/C/84dfea7961624d9baeee6e9124300099, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/f8160290180ba126c904acc32b60eac7/C/9cd152c0805f49fdace043f6485e1918, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/f8160290180ba126c904acc32b60eac7/C/3f1b6a9966784dbc8e885ce9ea199958, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/f8160290180ba126c904acc32b60eac7/C/11913411dbc84ae3a331aaf69d76e2ea, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/f8160290180ba126c904acc32b60eac7/C/6ca7e5c7d6c742f49e993eb5d01c790c] into tmpdir=hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/f8160290180ba126c904acc32b60eac7/.tmp, totalSize=70.2 K 2023-04-15 02:54:43,757 DEBUG [RS:0;jenkins-hbase20:34681-longCompactions-0] compactions.Compactor(207): Compacting 29656fb2fd6c4e998f291542df8d67fc, keycount=150, bloomtype=ROW, size=11.7 K, encoding=NONE, compression=NONE, seqNum=73, earliestPutTs=1681527279139 2023-04-15 02:54:43,757 DEBUG [RS:0;jenkins-hbase20:34681-longCompactions-0] compactions.Compactor(207): Compacting 84dfea7961624d9baeee6e9124300099, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=110, earliestPutTs=1681527279180 2023-04-15 02:54:43,758 DEBUG [RS:0;jenkins-hbase20:34681-longCompactions-0] compactions.Compactor(207): Compacting 9cd152c0805f49fdace043f6485e1918, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=122, earliestPutTs=1681527280288 2023-04-15 02:54:43,763 DEBUG [RS:0;jenkins-hbase20:34681-longCompactions-0] compactions.Compactor(207): Compacting 3f1b6a9966784dbc8e885ce9ea199958, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=158, earliestPutTs=1681527281416 2023-04-15 02:54:43,763 DEBUG [RS:0;jenkins-hbase20:34681-longCompactions-0] compactions.Compactor(207): Compacting 11913411dbc84ae3a331aaf69d76e2ea, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=168, earliestPutTs=1681527281981 2023-04-15 02:54:43,764 DEBUG [RS:0;jenkins-hbase20:34681-longCompactions-0] compactions.Compactor(207): Compacting 6ca7e5c7d6c742f49e993eb5d01c790c, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=178, earliestPutTs=1681527283131 2023-04-15 02:54:43,776 INFO [RS:0;jenkins-hbase20:34681-longCompactions-0] throttle.PressureAwareThroughputController(145): f8160290180ba126c904acc32b60eac7#C#compaction#212 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-15 02:54:43,819 DEBUG [RS:0;jenkins-hbase20:34681-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/f8160290180ba126c904acc32b60eac7/.tmp/C/041c340efc9c4b908be2bec6a35b7f72 as hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/f8160290180ba126c904acc32b60eac7/C/041c340efc9c4b908be2bec6a35b7f72 2023-04-15 02:54:43,827 INFO [RS:0;jenkins-hbase20:34681-longCompactions-0] regionserver.HStore(1652): Completed compaction of 6 (all) file(s) in f8160290180ba126c904acc32b60eac7/C of f8160290180ba126c904acc32b60eac7 into 041c340efc9c4b908be2bec6a35b7f72(size=12.1 K), total size for store is 12.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-15 02:54:43,828 DEBUG [RS:0;jenkins-hbase20:34681-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for f8160290180ba126c904acc32b60eac7: 2023-04-15 02:54:43,828 INFO [RS:0;jenkins-hbase20:34681-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681527274083.f8160290180ba126c904acc32b60eac7., storeName=f8160290180ba126c904acc32b60eac7/C, priority=10, startTime=1681527283624; duration=0sec 2023-04-15 02:54:43,828 DEBUG [RS:0;jenkins-hbase20:34681-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-15 02:54:43,907 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34681] regionserver.HRegion(9158): Flush requested on f8160290180ba126c904acc32b60eac7 2023-04-15 02:54:43,907 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681527274083.f8160290180ba126c904acc32b60eac7. as already flushing 2023-04-15 02:54:43,933 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f8160290180ba126c904acc32b60eac7, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:54:43,933 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34681] ipc.CallRunner(144): callId: 130 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:41918 deadline: 1681527343930, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f8160290180ba126c904acc32b60eac7, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:44,038 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f8160290180ba126c904acc32b60eac7, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:54:44,038 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34681] ipc.CallRunner(144): callId: 132 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:41918 deadline: 1681527344035, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f8160290180ba126c904acc32b60eac7, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:44,130 DEBUG [RS:0;jenkins-hbase20:34681-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/f8160290180ba126c904acc32b60eac7/.tmp/A/b8354584db9a4f43b0b4e9eef47e1892 as hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/f8160290180ba126c904acc32b60eac7/A/b8354584db9a4f43b0b4e9eef47e1892 2023-04-15 02:54:44,133 INFO [RS:0;jenkins-hbase20:34681-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 6 (all) file(s) in f8160290180ba126c904acc32b60eac7/A of f8160290180ba126c904acc32b60eac7 into b8354584db9a4f43b0b4e9eef47e1892(size=12.1 K), total size for store is 12.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-15 02:54:44,133 DEBUG [RS:0;jenkins-hbase20:34681-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for f8160290180ba126c904acc32b60eac7: 2023-04-15 02:54:44,134 INFO [RS:0;jenkins-hbase20:34681-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681527274083.f8160290180ba126c904acc32b60eac7., storeName=f8160290180ba126c904acc32b60eac7/A, priority=10, startTime=1681527283624; duration=0sec 2023-04-15 02:54:44,134 DEBUG [RS:0;jenkins-hbase20:34681-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-15 02:54:44,164 INFO [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=215 (bloomFilter=true), to=hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/f8160290180ba126c904acc32b60eac7/.tmp/A/954aa6d0069848788bf78dade2b6f2e7 2023-04-15 02:54:44,175 INFO [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=215 (bloomFilter=true), to=hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/f8160290180ba126c904acc32b60eac7/.tmp/B/70be9726a32642b48c6b33242243860b 2023-04-15 02:54:44,242 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f8160290180ba126c904acc32b60eac7, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:54:44,242 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34681] ipc.CallRunner(144): callId: 134 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:41918 deadline: 1681527344239, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f8160290180ba126c904acc32b60eac7, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:44,545 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f8160290180ba126c904acc32b60eac7, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:54:44,545 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34681] ipc.CallRunner(144): callId: 136 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:41918 deadline: 1681527344544, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f8160290180ba126c904acc32b60eac7, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:44,605 INFO [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=215 (bloomFilter=true), to=hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/f8160290180ba126c904acc32b60eac7/.tmp/C/01c51da9cec44adcb344482e733bcb14 2023-04-15 02:54:44,616 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/f8160290180ba126c904acc32b60eac7/.tmp/A/954aa6d0069848788bf78dade2b6f2e7 as hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/f8160290180ba126c904acc32b60eac7/A/954aa6d0069848788bf78dade2b6f2e7 2023-04-15 02:54:44,621 INFO [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/f8160290180ba126c904acc32b60eac7/A/954aa6d0069848788bf78dade2b6f2e7, entries=150, sequenceid=215, filesize=11.8 K 2023-04-15 02:54:44,621 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/f8160290180ba126c904acc32b60eac7/.tmp/B/70be9726a32642b48c6b33242243860b as hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/f8160290180ba126c904acc32b60eac7/B/70be9726a32642b48c6b33242243860b 2023-04-15 02:54:44,635 INFO [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/f8160290180ba126c904acc32b60eac7/B/70be9726a32642b48c6b33242243860b, entries=150, sequenceid=215, filesize=11.8 K 2023-04-15 02:54:44,636 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/f8160290180ba126c904acc32b60eac7/.tmp/C/01c51da9cec44adcb344482e733bcb14 as hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/f8160290180ba126c904acc32b60eac7/C/01c51da9cec44adcb344482e733bcb14 2023-04-15 02:54:44,647 INFO [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/f8160290180ba126c904acc32b60eac7/C/01c51da9cec44adcb344482e733bcb14, entries=150, sequenceid=215, filesize=11.8 K 2023-04-15 02:54:44,648 INFO [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~134.18 KB/137400, heapSize ~384.94 KB/394176, currentSize=46.96 KB/48090 for f8160290180ba126c904acc32b60eac7 in 930ms, sequenceid=215, compaction requested=false 2023-04-15 02:54:44,648 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for f8160290180ba126c904acc32b60eac7: 2023-04-15 02:54:44,648 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681527274083.f8160290180ba126c904acc32b60eac7. 2023-04-15 02:54:44,648 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-15 02:54:44,648 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-15 02:54:44,648 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:44,648 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-15 02:54:44,648 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34681,1681527209133' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-15 02:54:44,649 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:44,649 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:44,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:44,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:54:44,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:54:44,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:54:44,650 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:44,650 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-15 02:54:44,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:54:44,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:54:44,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:44,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:54:44,652 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34681,1681527209133' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-15 02:54:44,652 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1c6fc3e5[Count = 0] remaining members to acquire global barrier 2023-04-15 02:54:44,652 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-15 02:54:44,652 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:44,653 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:44,653 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:44,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:44,653 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-15 02:54:44,653 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-15 02:54:44,653 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34681,1681527209133' in zk 2023-04-15 02:54:44,653 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:44,653 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-15 02:54:44,654 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:44,654 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-15 02:54:44,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:44,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:54:44,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:54:44,654 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-15 02:54:44,654 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-15 02:54:44,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:54:44,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:54:44,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:54:44,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:44,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:54:44,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:54:44,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:44,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34681,1681527209133': 2023-04-15 02:54:44,658 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34681,1681527209133' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-15 02:54:44,658 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-15 02:54:44,658 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-15 02:54:44,658 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-15 02:54:44,658 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-15 02:54:44,658 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-15 02:54:44,666 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:44,666 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-15 02:54:44,666 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:44,666 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:44,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:44,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:44,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:54:44,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:54:44,666 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-15 02:54:44,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-15 02:54:44,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:54:44,667 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:44,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:54:44,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:44,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:54:44,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:54:44,668 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:44,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:44,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:54:44,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:54:44,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:44,670 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-15 02:54:44,670 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-15 02:54:44,670 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-15 02:54:44,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-15 02:54:44,670 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-15 02:54:44,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-15 02:54:44,671 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:44,671 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:54:44,671 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:54:44,671 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:44,671 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:44,671 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:44,671 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:44,671 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-15 02:54:44,671 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-15 02:54:44,671 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-15 02:54:44,671 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-15 02:54:44,672 DEBUG [Thread-1527] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-15 02:54:44,672 DEBUG [Thread-1527] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-15 02:54:44,772 DEBUG [Thread-1527] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-15 02:54:44,772 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-15 02:54:44,774 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-15 02:54:44,777 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-15 02:54:44,777 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-15 02:54:44,777 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-15 02:54:44,777 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-15 02:54:44,777 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-15 02:54:44,777 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-15 02:54:44,778 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:44,778 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:54:44,778 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:44,778 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-15 02:54:44,778 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-15 02:54:44,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-15 02:54:44,779 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:44,779 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-15 02:54:44,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:54:44,779 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:44,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-15 02:54:44,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:54:44,779 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-15 02:54:44,780 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-15 02:54:44,780 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-15 02:54:44,780 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-15 02:54:44,780 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-15 02:54:44,780 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-15 02:54:44,780 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-15 02:54:44,780 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681527274083.f8160290180ba126c904acc32b60eac7. 2023-04-15 02:54:44,780 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681527274083.f8160290180ba126c904acc32b60eac7. started... 2023-04-15 02:54:44,780 INFO [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing f8160290180ba126c904acc32b60eac7 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-15 02:54:44,780 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK f8160290180ba126c904acc32b60eac7, store=A 2023-04-15 02:54:44,780 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-15 02:54:44,780 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK f8160290180ba126c904acc32b60eac7, store=B 2023-04-15 02:54:44,781 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-15 02:54:44,781 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK f8160290180ba126c904acc32b60eac7, store=C 2023-04-15 02:54:44,781 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-15 02:54:45,052 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34681] regionserver.HRegion(9158): Flush requested on f8160290180ba126c904acc32b60eac7 2023-04-15 02:54:45,052 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681527274083.f8160290180ba126c904acc32b60eac7. as already flushing 2023-04-15 02:54:45,097 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681527285084, max timestamp=1681527285096Num uniques 859; 2023-04-15 02:54:45,097 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-6] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-15 02:54:45,097 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681527285084, max timestamp=1681527285096Num uniques 859; 2023-04-15 02:54:45,097 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-7] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-15 02:54:45,097 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681527285084, max timestamp=1681527285096Num uniques 859; 2023-04-15 02:54:45,098 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-15 02:54:45,166 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f8160290180ba126c904acc32b60eac7, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:54:45,166 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34681] ipc.CallRunner(144): callId: 172 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:41918 deadline: 1681527345165, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f8160290180ba126c904acc32b60eac7, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:45,192 INFO [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=228 (bloomFilter=true), to=hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/f8160290180ba126c904acc32b60eac7/.tmp/A/6846a5a1400b41fb908682d63eb5d93e 2023-04-15 02:54:45,268 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f8160290180ba126c904acc32b60eac7, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:54:45,268 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34681] ipc.CallRunner(144): callId: 174 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:41918 deadline: 1681527345267, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f8160290180ba126c904acc32b60eac7, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:45,471 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f8160290180ba126c904acc32b60eac7, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:54:45,471 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34681] ipc.CallRunner(144): callId: 176 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:41918 deadline: 1681527345469, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f8160290180ba126c904acc32b60eac7, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:45,561 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f8160290180ba126c904acc32b60eac7, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:54:45,561 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34681] ipc.CallRunner(144): callId: 71 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41890 deadline: 1681527345560, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f8160290180ba126c904acc32b60eac7, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:45,571 DEBUG [Thread-1517] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4145 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f8160290180ba126c904acc32b60eac7, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_0' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1681527274083.f8160290180ba126c904acc32b60eac7., hostname=jenkins-hbase20.apache.org,34681,1681527209133, seqNum=2, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f8160290180ba126c904acc32b60eac7, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor31.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f8160290180ba126c904acc32b60eac7, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-15 02:54:45,572 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f8160290180ba126c904acc32b60eac7, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:54:45,573 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34681] ipc.CallRunner(144): callId: 64 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41940 deadline: 1681527345572, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f8160290180ba126c904acc32b60eac7, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:45,574 DEBUG [Thread-1521] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4160 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f8160290180ba126c904acc32b60eac7, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_2' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1681527274083.f8160290180ba126c904acc32b60eac7., hostname=jenkins-hbase20.apache.org,34681,1681527209133, seqNum=2, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f8160290180ba126c904acc32b60eac7, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor31.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f8160290180ba126c904acc32b60eac7, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-15 02:54:45,580 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f8160290180ba126c904acc32b60eac7, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:54:45,580 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34681] ipc.CallRunner(144): callId: 65 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41916 deadline: 1681527345579, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f8160290180ba126c904acc32b60eac7, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:45,582 DEBUG [Thread-1523] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4163 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f8160290180ba126c904acc32b60eac7, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_1' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1681527274083.f8160290180ba126c904acc32b60eac7., hostname=jenkins-hbase20.apache.org,34681,1681527209133, seqNum=2, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f8160290180ba126c904acc32b60eac7, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor31.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f8160290180ba126c904acc32b60eac7, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-15 02:54:45,583 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f8160290180ba126c904acc32b60eac7, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:54:45,583 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34681] ipc.CallRunner(144): callId: 75 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41926 deadline: 1681527345580, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f8160290180ba126c904acc32b60eac7, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:45,584 DEBUG [Thread-1525] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4165 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f8160290180ba126c904acc32b60eac7, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_1' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1681527274083.f8160290180ba126c904acc32b60eac7., hostname=jenkins-hbase20.apache.org,34681,1681527209133, seqNum=2, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f8160290180ba126c904acc32b60eac7, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor31.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f8160290180ba126c904acc32b60eac7, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-15 02:54:45,623 INFO [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=228 (bloomFilter=true), to=hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/f8160290180ba126c904acc32b60eac7/.tmp/B/e8e29ea90fdc4fdeb8353ff44c9293d2 2023-04-15 02:54:45,774 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f8160290180ba126c904acc32b60eac7, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:54:45,774 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34681] ipc.CallRunner(144): callId: 178 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:41918 deadline: 1681527345773, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f8160290180ba126c904acc32b60eac7, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:46,050 INFO [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=228 (bloomFilter=true), to=hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/f8160290180ba126c904acc32b60eac7/.tmp/C/3a07d2aaddaa4447a6531f316856f1cc 2023-04-15 02:54:46,055 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/f8160290180ba126c904acc32b60eac7/.tmp/A/6846a5a1400b41fb908682d63eb5d93e as hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/f8160290180ba126c904acc32b60eac7/A/6846a5a1400b41fb908682d63eb5d93e 2023-04-15 02:54:46,060 INFO [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/f8160290180ba126c904acc32b60eac7/A/6846a5a1400b41fb908682d63eb5d93e, entries=100, sequenceid=228, filesize=9.4 K 2023-04-15 02:54:46,061 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/f8160290180ba126c904acc32b60eac7/.tmp/B/e8e29ea90fdc4fdeb8353ff44c9293d2 as hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/f8160290180ba126c904acc32b60eac7/B/e8e29ea90fdc4fdeb8353ff44c9293d2 2023-04-15 02:54:46,068 INFO [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/f8160290180ba126c904acc32b60eac7/B/e8e29ea90fdc4fdeb8353ff44c9293d2, entries=100, sequenceid=228, filesize=9.4 K 2023-04-15 02:54:46,069 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/f8160290180ba126c904acc32b60eac7/.tmp/C/3a07d2aaddaa4447a6531f316856f1cc as hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/f8160290180ba126c904acc32b60eac7/C/3a07d2aaddaa4447a6531f316856f1cc 2023-04-15 02:54:46,073 INFO [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/f8160290180ba126c904acc32b60eac7/C/3a07d2aaddaa4447a6531f316856f1cc, entries=100, sequenceid=228, filesize=9.4 K 2023-04-15 02:54:46,074 INFO [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=134.18 KB/137400 for f8160290180ba126c904acc32b60eac7 in 1294ms, sequenceid=228, compaction requested=true 2023-04-15 02:54:46,074 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for f8160290180ba126c904acc32b60eac7: 2023-04-15 02:54:46,074 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681527274083.f8160290180ba126c904acc32b60eac7. 2023-04-15 02:54:46,074 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-15 02:54:46,074 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-15 02:54:46,074 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:46,074 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-15 02:54:46,075 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34681,1681527209133' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-15 02:54:46,075 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:46,075 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:46,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:46,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:54:46,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:54:46,076 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:46,076 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-15 02:54:46,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:54:46,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:54:46,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:54:46,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:46,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:54:46,077 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34681,1681527209133' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-15 02:54:46,077 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@262a7234[Count = 0] remaining members to acquire global barrier 2023-04-15 02:54:46,077 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-15 02:54:46,077 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:46,079 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:46,079 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:46,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:46,079 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-15 02:54:46,079 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-15 02:54:46,080 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34681,1681527209133' in zk 2023-04-15 02:54:46,080 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:46,080 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-15 02:54:46,080 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-15 02:54:46,080 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-15 02:54:46,080 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:46,080 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-15 02:54:46,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:46,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:54:46,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:54:46,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:54:46,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:54:46,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:54:46,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:46,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:54:46,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:54:46,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:46,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34681,1681527209133': 2023-04-15 02:54:46,083 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34681,1681527209133' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-15 02:54:46,083 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-15 02:54:46,083 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-15 02:54:46,083 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-15 02:54:46,083 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-15 02:54:46,083 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-15 02:54:46,084 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:46,084 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-15 02:54:46,084 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:46,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:46,084 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:46,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:46,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:54:46,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:54:46,084 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-15 02:54:46,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-15 02:54:46,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:46,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:54:46,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:54:46,086 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:46,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:54:46,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:54:46,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:46,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:54:46,087 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:46,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:54:46,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:46,089 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-15 02:54:46,089 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-15 02:54:46,089 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:46,089 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-15 02:54:46,089 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-15 02:54:46,089 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-15 02:54:46,089 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-15 02:54:46,089 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-15 02:54:46,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-15 02:54:46,089 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:54:46,089 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:54:46,089 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:46,089 DEBUG [Thread-1527] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-15 02:54:46,089 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:46,089 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:46,089 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:46,089 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-15 02:54:46,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-15 02:54:46,089 DEBUG [Thread-1527] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-15 02:54:46,190 DEBUG [Thread-1527] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-15 02:54:46,190 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-15 02:54:46,192 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-15 02:54:46,195 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-15 02:54:46,195 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-15 02:54:46,195 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-15 02:54:46,195 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-15 02:54:46,195 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-15 02:54:46,195 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-15 02:54:46,196 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:46,196 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:54:46,196 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:46,196 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-15 02:54:46,196 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-15 02:54:46,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-15 02:54:46,197 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:46,197 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-15 02:54:46,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:54:46,197 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:46,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-15 02:54:46,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:54:46,197 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-15 02:54:46,197 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-15 02:54:46,197 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-15 02:54:46,197 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-15 02:54:46,198 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-15 02:54:46,198 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-15 02:54:46,198 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-15 02:54:46,198 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-15 02:54:46,198 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681527274083.f8160290180ba126c904acc32b60eac7. 2023-04-15 02:54:46,198 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681527274083.f8160290180ba126c904acc32b60eac7. started... 2023-04-15 02:54:46,198 INFO [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing f8160290180ba126c904acc32b60eac7 3/3 column families, dataSize=134.18 KB heapSize=384.98 KB 2023-04-15 02:54:46,198 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK f8160290180ba126c904acc32b60eac7, store=A 2023-04-15 02:54:46,198 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-15 02:54:46,198 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK f8160290180ba126c904acc32b60eac7, store=B 2023-04-15 02:54:46,198 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-15 02:54:46,199 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK f8160290180ba126c904acc32b60eac7, store=C 2023-04-15 02:54:46,199 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-15 02:54:46,212 INFO [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=265 (bloomFilter=true), to=hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/f8160290180ba126c904acc32b60eac7/.tmp/A/9276f313bec440ab82008fb21fe3271a 2023-04-15 02:54:46,224 INFO [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=265 (bloomFilter=true), to=hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/f8160290180ba126c904acc32b60eac7/.tmp/B/f035898cd29a4386814b629068363bef 2023-04-15 02:54:46,276 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34681] regionserver.HRegion(9158): Flush requested on f8160290180ba126c904acc32b60eac7 2023-04-15 02:54:46,276 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681527274083.f8160290180ba126c904acc32b60eac7. as already flushing 2023-04-15 02:54:46,296 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f8160290180ba126c904acc32b60eac7, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:54:46,297 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34681] ipc.CallRunner(144): callId: 187 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:41918 deadline: 1681527346294, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f8160290180ba126c904acc32b60eac7, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:46,399 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f8160290180ba126c904acc32b60eac7, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:54:46,399 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34681] ipc.CallRunner(144): callId: 189 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:41918 deadline: 1681527346397, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f8160290180ba126c904acc32b60eac7, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:46,603 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f8160290180ba126c904acc32b60eac7, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:54:46,604 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34681] ipc.CallRunner(144): callId: 191 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:41918 deadline: 1681527346601, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f8160290180ba126c904acc32b60eac7, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:46,636 INFO [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=265 (bloomFilter=true), to=hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/f8160290180ba126c904acc32b60eac7/.tmp/C/9b2876c82e6d44f0b51ff6f305538d23 2023-04-15 02:54:46,640 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/f8160290180ba126c904acc32b60eac7/.tmp/A/9276f313bec440ab82008fb21fe3271a as hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/f8160290180ba126c904acc32b60eac7/A/9276f313bec440ab82008fb21fe3271a 2023-04-15 02:54:46,644 INFO [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/f8160290180ba126c904acc32b60eac7/A/9276f313bec440ab82008fb21fe3271a, entries=150, sequenceid=265, filesize=11.9 K 2023-04-15 02:54:46,645 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/f8160290180ba126c904acc32b60eac7/.tmp/B/f035898cd29a4386814b629068363bef as hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/f8160290180ba126c904acc32b60eac7/B/f035898cd29a4386814b629068363bef 2023-04-15 02:54:46,648 INFO [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/f8160290180ba126c904acc32b60eac7/B/f035898cd29a4386814b629068363bef, entries=150, sequenceid=265, filesize=11.9 K 2023-04-15 02:54:46,649 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/f8160290180ba126c904acc32b60eac7/.tmp/C/9b2876c82e6d44f0b51ff6f305538d23 as hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/f8160290180ba126c904acc32b60eac7/C/9b2876c82e6d44f0b51ff6f305538d23 2023-04-15 02:54:46,653 INFO [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/f8160290180ba126c904acc32b60eac7/C/9b2876c82e6d44f0b51ff6f305538d23, entries=150, sequenceid=265, filesize=11.9 K 2023-04-15 02:54:46,654 INFO [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~134.18 KB/137400, heapSize ~384.94 KB/394176, currentSize=46.96 KB/48090 for f8160290180ba126c904acc32b60eac7 in 456ms, sequenceid=265, compaction requested=true 2023-04-15 02:54:46,654 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for f8160290180ba126c904acc32b60eac7: 2023-04-15 02:54:46,654 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681527274083.f8160290180ba126c904acc32b60eac7. 2023-04-15 02:54:46,654 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-15 02:54:46,654 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-15 02:54:46,654 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:46,654 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-15 02:54:46,654 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34681,1681527209133' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-15 02:54:46,655 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:46,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:46,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:54:46,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:54:46,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:54:46,656 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:46,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:54:46,656 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:46,656 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-15 02:54:46,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:54:46,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:46,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:54:46,658 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34681,1681527209133' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-15 02:54:46,658 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@26e2c43c[Count = 0] remaining members to acquire global barrier 2023-04-15 02:54:46,658 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-15 02:54:46,658 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:46,658 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:46,658 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:46,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:46,658 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-15 02:54:46,659 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-15 02:54:46,659 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34681,1681527209133' in zk 2023-04-15 02:54:46,659 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:46,659 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-15 02:54:46,659 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-15 02:54:46,659 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-15 02:54:46,659 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-15 02:54:46,660 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:46,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:46,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:54:46,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:54:46,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:54:46,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:54:46,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:54:46,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:46,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:54:46,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:54:46,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:46,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34681,1681527209133': 2023-04-15 02:54:46,663 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34681,1681527209133' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-15 02:54:46,663 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-15 02:54:46,663 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-15 02:54:46,663 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-15 02:54:46,663 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-15 02:54:46,663 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-15 02:54:46,664 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:46,664 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-15 02:54:46,664 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:46,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:46,664 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:46,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:46,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:54:46,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:54:46,664 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-15 02:54:46,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-15 02:54:46,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:54:46,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:46,665 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:46,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:54:46,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:54:46,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:54:46,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:46,666 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:46,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:54:46,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:54:46,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:46,667 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:46,667 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:54:46,667 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:54:46,668 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:46,668 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:46,668 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-15 02:54:46,668 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-15 02:54:46,668 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-15 02:54:46,668 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-15 02:54:46,668 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-15 02:54:46,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-15 02:54:46,668 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:46,668 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:46,668 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-15 02:54:46,668 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-15 02:54:46,668 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-15 02:54:46,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-15 02:54:46,668 DEBUG [Thread-1527] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-15 02:54:46,668 DEBUG [Thread-1527] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-15 02:54:46,768 DEBUG [Thread-1527] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-15 02:54:46,769 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-15 02:54:46,770 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-15 02:54:46,776 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-15 02:54:46,777 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-15 02:54:46,777 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-15 02:54:46,777 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-15 02:54:46,777 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-15 02:54:46,777 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-15 02:54:46,777 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:46,777 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:54:46,778 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-15 02:54:46,778 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:46,778 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-15 02:54:46,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-15 02:54:46,778 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:46,778 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-15 02:54:46,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:54:46,779 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:46,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-15 02:54:46,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:54:46,779 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-15 02:54:46,779 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-15 02:54:46,779 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-15 02:54:46,779 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-15 02:54:46,779 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-15 02:54:46,780 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-15 02:54:46,780 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-15 02:54:46,780 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-15 02:54:46,780 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681527274083.f8160290180ba126c904acc32b60eac7. 2023-04-15 02:54:46,780 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681527274083.f8160290180ba126c904acc32b60eac7. started... 2023-04-15 02:54:46,780 INFO [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing f8160290180ba126c904acc32b60eac7 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-15 02:54:46,780 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK f8160290180ba126c904acc32b60eac7, store=A 2023-04-15 02:54:46,780 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-15 02:54:46,780 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK f8160290180ba126c904acc32b60eac7, store=B 2023-04-15 02:54:46,780 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-15 02:54:46,780 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK f8160290180ba126c904acc32b60eac7, store=C 2023-04-15 02:54:46,780 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-15 02:54:46,907 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34681] regionserver.HRegion(9158): Flush requested on f8160290180ba126c904acc32b60eac7 2023-04-15 02:54:46,907 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681527274083.f8160290180ba126c904acc32b60eac7. as already flushing 2023-04-15 02:54:46,984 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681527286955, max timestamp=1681527286983Num uniques 859; 2023-04-15 02:54:46,984 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-8] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-15 02:54:46,984 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681527286955, max timestamp=1681527286983Num uniques 859; 2023-04-15 02:54:46,985 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-2] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-15 02:54:46,985 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681527286955, max timestamp=1681527286983Num uniques 859; 2023-04-15 02:54:46,985 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-9] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-15 02:54:47,070 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f8160290180ba126c904acc32b60eac7, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:54:47,070 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34681] ipc.CallRunner(144): callId: 227 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:41918 deadline: 1681527347066, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f8160290180ba126c904acc32b60eac7, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:47,173 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f8160290180ba126c904acc32b60eac7, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:54:47,173 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34681] ipc.CallRunner(144): callId: 229 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:41918 deadline: 1681527347171, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f8160290180ba126c904acc32b60eac7, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:47,216 INFO [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=275 (bloomFilter=true), to=hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/f8160290180ba126c904acc32b60eac7/.tmp/A/774c69fb42ef48febac3611cacd2e05f 2023-04-15 02:54:47,376 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f8160290180ba126c904acc32b60eac7, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:54:47,376 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34681] ipc.CallRunner(144): callId: 231 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:41918 deadline: 1681527347374, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f8160290180ba126c904acc32b60eac7, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:47,628 INFO [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=275 (bloomFilter=true), to=hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/f8160290180ba126c904acc32b60eac7/.tmp/B/1af10683cc6a452aaece1b4f85a3c87b 2023-04-15 02:54:47,680 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f8160290180ba126c904acc32b60eac7, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:54:47,681 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34681] ipc.CallRunner(144): callId: 233 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:41918 deadline: 1681527347679, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f8160290180ba126c904acc32b60eac7, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:48,060 INFO [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=275 (bloomFilter=true), to=hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/f8160290180ba126c904acc32b60eac7/.tmp/C/7807ee8f8b164f238aca4f3363e0ee36 2023-04-15 02:54:48,065 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/f8160290180ba126c904acc32b60eac7/.tmp/A/774c69fb42ef48febac3611cacd2e05f as hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/f8160290180ba126c904acc32b60eac7/A/774c69fb42ef48febac3611cacd2e05f 2023-04-15 02:54:48,070 INFO [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/f8160290180ba126c904acc32b60eac7/A/774c69fb42ef48febac3611cacd2e05f, entries=150, sequenceid=275, filesize=11.9 K 2023-04-15 02:54:48,071 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/f8160290180ba126c904acc32b60eac7/.tmp/B/1af10683cc6a452aaece1b4f85a3c87b as hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/f8160290180ba126c904acc32b60eac7/B/1af10683cc6a452aaece1b4f85a3c87b 2023-04-15 02:54:48,075 INFO [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/f8160290180ba126c904acc32b60eac7/B/1af10683cc6a452aaece1b4f85a3c87b, entries=150, sequenceid=275, filesize=11.9 K 2023-04-15 02:54:48,076 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/f8160290180ba126c904acc32b60eac7/.tmp/C/7807ee8f8b164f238aca4f3363e0ee36 as hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/f8160290180ba126c904acc32b60eac7/C/7807ee8f8b164f238aca4f3363e0ee36 2023-04-15 02:54:48,085 INFO [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/f8160290180ba126c904acc32b60eac7/C/7807ee8f8b164f238aca4f3363e0ee36, entries=150, sequenceid=275, filesize=11.9 K 2023-04-15 02:54:48,086 INFO [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=134.18 KB/137400 for f8160290180ba126c904acc32b60eac7 in 1306ms, sequenceid=275, compaction requested=true 2023-04-15 02:54:48,086 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for f8160290180ba126c904acc32b60eac7: 2023-04-15 02:54:48,086 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681527274083.f8160290180ba126c904acc32b60eac7. 2023-04-15 02:54:48,086 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-15 02:54:48,086 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-15 02:54:48,086 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:48,086 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-15 02:54:48,087 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34681,1681527209133' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-15 02:54:48,087 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:48,087 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:48,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:48,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:54:48,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:54:48,088 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:48,088 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-15 02:54:48,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:54:48,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:54:48,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:54:48,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:48,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:54:48,089 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34681,1681527209133' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-15 02:54:48,089 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@e90f5a7[Count = 0] remaining members to acquire global barrier 2023-04-15 02:54:48,089 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-15 02:54:48,089 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:48,089 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:48,089 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:48,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:48,090 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-15 02:54:48,090 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-15 02:54:48,090 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34681,1681527209133' in zk 2023-04-15 02:54:48,090 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:48,090 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-15 02:54:48,091 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:48,091 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-15 02:54:48,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:48,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:54:48,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:54:48,091 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-15 02:54:48,091 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-15 02:54:48,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:54:48,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:54:48,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:54:48,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:48,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:54:48,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:54:48,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:48,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34681,1681527209133': 2023-04-15 02:54:48,094 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34681,1681527209133' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-15 02:54:48,094 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-15 02:54:48,094 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-15 02:54:48,094 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-15 02:54:48,094 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-15 02:54:48,094 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-15 02:54:48,095 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:48,095 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-15 02:54:48,095 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:48,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:48,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:54:48,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:54:48,095 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:48,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:48,099 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-15 02:54:48,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-15 02:54:48,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:54:48,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:48,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:54:48,100 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:48,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:54:48,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:54:48,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:48,102 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:48,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:54:48,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:54:48,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:48,112 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-15 02:54:48,112 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-15 02:54:48,112 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-15 02:54:48,112 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-15 02:54:48,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-15 02:54:48,112 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:48,112 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-15 02:54:48,112 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-15 02:54:48,112 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:54:48,113 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:54:48,113 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:48,113 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:48,113 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:48,113 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:48,113 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-15 02:54:48,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-15 02:54:48,113 DEBUG [Thread-1527] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-15 02:54:48,113 DEBUG [Thread-1527] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-15 02:54:48,185 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34681] regionserver.HRegion(9158): Flush requested on f8160290180ba126c904acc32b60eac7 2023-04-15 02:54:48,185 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing f8160290180ba126c904acc32b60eac7 3/3 column families, dataSize=140.89 KB heapSize=405.61 KB 2023-04-15 02:54:48,185 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK f8160290180ba126c904acc32b60eac7, store=A 2023-04-15 02:54:48,185 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=3, new segment=null 2023-04-15 02:54:48,185 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK f8160290180ba126c904acc32b60eac7, store=B 2023-04-15 02:54:48,185 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=3, new segment=null 2023-04-15 02:54:48,185 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK f8160290180ba126c904acc32b60eac7, store=C 2023-04-15 02:54:48,185 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=3, new segment=null 2023-04-15 02:54:48,213 DEBUG [Thread-1527] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-15 02:54:48,213 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-15 02:54:48,215 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-15 02:54:48,218 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-15 02:54:48,218 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-15 02:54:48,218 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-15 02:54:48,218 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-15 02:54:48,218 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-15 02:54:48,218 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-15 02:54:48,218 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f8160290180ba126c904acc32b60eac7, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:54:48,218 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34681] ipc.CallRunner(144): callId: 242 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:41918 deadline: 1681527348216, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f8160290180ba126c904acc32b60eac7, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:48,226 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:48,226 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:54:48,228 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-15 02:54:48,228 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:48,228 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-15 02:54:48,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-15 02:54:48,228 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:48,228 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-15 02:54:48,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:54:48,228 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:48,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-15 02:54:48,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:54:48,229 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-15 02:54:48,229 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-15 02:54:48,229 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-15 02:54:48,229 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-15 02:54:48,229 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-15 02:54:48,230 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-15 02:54:48,230 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-15 02:54:48,230 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681527274083.f8160290180ba126c904acc32b60eac7. 2023-04-15 02:54:48,230 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681527274083.f8160290180ba126c904acc32b60eac7. started... 2023-04-15 02:54:48,230 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681527274083.f8160290180ba126c904acc32b60eac7. as already flushing 2023-04-15 02:54:48,230 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681527274083.f8160290180ba126c904acc32b60eac7. 2023-04-15 02:54:48,230 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-15 02:54:48,230 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-15 02:54:48,230 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:48,230 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-15 02:54:48,230 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34681,1681527209133' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-15 02:54:48,238 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:48,238 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:48,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:48,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:54:48,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:54:48,238 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:48,238 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-15 02:54:48,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:54:48,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:54:48,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:54:48,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:48,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:54:48,240 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34681,1681527209133' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-15 02:54:48,240 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6c826f7b[Count = 0] remaining members to acquire global barrier 2023-04-15 02:54:48,240 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-15 02:54:48,240 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:48,245 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:48,245 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:48,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:48,246 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-15 02:54:48,246 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-15 02:54:48,246 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34681,1681527209133' in zk 2023-04-15 02:54:48,246 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:48,246 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-15 02:54:48,251 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-15 02:54:48,251 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-15 02:54:48,251 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-15 02:54:48,251 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:48,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:48,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:54:48,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:54:48,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:54:48,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:54:48,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:54:48,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:48,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:54:48,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:54:48,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:48,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34681,1681527209133': 2023-04-15 02:54:48,261 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34681,1681527209133' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-15 02:54:48,261 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-15 02:54:48,261 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-15 02:54:48,261 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-15 02:54:48,261 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-15 02:54:48,261 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-15 02:54:48,270 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:48,270 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-15 02:54:48,270 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:48,270 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:48,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:48,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:54:48,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:54:48,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:48,271 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-15 02:54:48,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-15 02:54:48,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:54:48,271 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:48,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:48,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:54:48,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:54:48,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:54:48,272 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:48,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:48,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:54:48,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:54:48,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:48,274 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-15 02:54:48,274 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:48,274 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-15 02:54:48,274 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:54:48,274 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-15 02:54:48,274 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-15 02:54:48,274 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-15 02:54:48,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-15 02:54:48,274 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-15 02:54:48,274 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-15 02:54:48,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-15 02:54:48,274 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-15 02:54:48,274 DEBUG [Thread-1527] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-15 02:54:48,274 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:54:48,275 DEBUG [Thread-1527] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-15 02:54:48,275 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:48,275 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:48,275 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:48,275 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:48,323 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f8160290180ba126c904acc32b60eac7, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:54:48,323 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34681] ipc.CallRunner(144): callId: 244 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:41918 deadline: 1681527348319, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f8160290180ba126c904acc32b60eac7, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:48,375 DEBUG [Thread-1527] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-15 02:54:48,375 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-15 02:54:48,377 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-15 02:54:48,379 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-15 02:54:48,379 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-15 02:54:48,379 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-15 02:54:48,379 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-15 02:54:48,379 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-15 02:54:48,379 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-15 02:54:48,380 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:48,380 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:54:48,380 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-15 02:54:48,380 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:48,380 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-15 02:54:48,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-15 02:54:48,381 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:48,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:54:48,381 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-15 02:54:48,381 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:48,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-15 02:54:48,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:54:48,381 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-15 02:54:48,381 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-15 02:54:48,381 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-15 02:54:48,382 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-15 02:54:48,382 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-15 02:54:48,382 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-15 02:54:48,382 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-15 02:54:48,382 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-15 02:54:48,382 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681527274083.f8160290180ba126c904acc32b60eac7. 2023-04-15 02:54:48,382 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681527274083.f8160290180ba126c904acc32b60eac7. started... 2023-04-15 02:54:48,382 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681527274083.f8160290180ba126c904acc32b60eac7. as already flushing 2023-04-15 02:54:48,382 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681527274083.f8160290180ba126c904acc32b60eac7. 2023-04-15 02:54:48,382 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-15 02:54:48,382 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-15 02:54:48,382 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:48,382 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-15 02:54:48,382 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34681,1681527209133' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-15 02:54:48,383 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:48,383 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:48,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:48,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:54:48,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:54:48,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:54:48,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:54:48,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:54:48,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:48,384 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:48,384 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-15 02:54:48,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:54:48,385 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34681,1681527209133' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-15 02:54:48,385 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@19cc0b0d[Count = 0] remaining members to acquire global barrier 2023-04-15 02:54:48,385 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-15 02:54:48,385 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:48,385 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:48,386 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:48,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:48,386 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-15 02:54:48,386 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-15 02:54:48,386 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34681,1681527209133' in zk 2023-04-15 02:54:48,386 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:48,386 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-15 02:54:48,386 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-15 02:54:48,386 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-15 02:54:48,387 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-15 02:54:48,386 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:48,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:48,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:54:48,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:54:48,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:54:48,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:54:48,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:54:48,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:48,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:54:48,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:54:48,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:48,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34681,1681527209133': 2023-04-15 02:54:48,390 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34681,1681527209133' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-15 02:54:48,390 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-15 02:54:48,390 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-15 02:54:48,390 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-15 02:54:48,390 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-15 02:54:48,390 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-15 02:54:48,399 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:48,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:48,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:54:48,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:54:48,399 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:48,400 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-15 02:54:48,400 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:48,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:48,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:54:48,400 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:48,400 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-15 02:54:48,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-15 02:54:48,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:54:48,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:48,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:54:48,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:54:48,402 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:48,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:48,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:54:48,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:54:48,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:48,407 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-15 02:54:48,408 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-15 02:54:48,408 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-15 02:54:48,408 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:48,408 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-15 02:54:48,408 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-15 02:54:48,408 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:54:48,408 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-15 02:54:48,408 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:54:48,408 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-15 02:54:48,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-15 02:54:48,408 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:48,408 DEBUG [Thread-1527] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-15 02:54:48,408 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:48,408 DEBUG [Thread-1527] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-15 02:54:48,408 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:48,408 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:48,408 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-15 02:54:48,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-15 02:54:48,508 DEBUG [Thread-1527] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-15 02:54:48,509 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-15 02:54:48,510 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-15 02:54:48,511 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-15 02:54:48,511 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-15 02:54:48,511 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-15 02:54:48,511 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-15 02:54:48,511 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-15 02:54:48,511 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-15 02:54:48,511 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:48,511 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:54:48,512 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-15 02:54:48,512 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:48,512 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-15 02:54:48,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-15 02:54:48,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:54:48,512 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:48,512 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-15 02:54:48,513 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:48,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-15 02:54:48,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:54:48,513 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-15 02:54:48,513 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-15 02:54:48,513 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-15 02:54:48,513 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-15 02:54:48,513 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-15 02:54:48,513 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-15 02:54:48,517 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-15 02:54:48,517 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-15 02:54:48,517 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681527274083.f8160290180ba126c904acc32b60eac7. 2023-04-15 02:54:48,517 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681527274083.f8160290180ba126c904acc32b60eac7. started... 2023-04-15 02:54:48,517 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681527274083.f8160290180ba126c904acc32b60eac7. as already flushing 2023-04-15 02:54:48,517 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681527274083.f8160290180ba126c904acc32b60eac7. 2023-04-15 02:54:48,517 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-15 02:54:48,517 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-15 02:54:48,517 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:48,517 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-15 02:54:48,517 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34681,1681527209133' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-15 02:54:48,518 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:48,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:48,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:54:48,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:54:48,518 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:48,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:54:48,518 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:48,518 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-15 02:54:48,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:54:48,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:54:48,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:48,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:54:48,519 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34681,1681527209133' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-15 02:54:48,519 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@293fff2c[Count = 0] remaining members to acquire global barrier 2023-04-15 02:54:48,519 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-15 02:54:48,519 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:48,519 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:48,519 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:48,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:48,520 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-15 02:54:48,520 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-15 02:54:48,520 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:48,520 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-15 02:54:48,520 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34681,1681527209133' in zk 2023-04-15 02:54:48,520 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:48,520 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-15 02:54:48,520 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-15 02:54:48,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:48,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:54:48,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:54:48,520 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-15 02:54:48,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:54:48,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:54:48,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:54:48,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:48,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:54:48,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:54:48,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:48,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34681,1681527209133': 2023-04-15 02:54:48,522 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34681,1681527209133' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-15 02:54:48,522 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-15 02:54:48,522 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-15 02:54:48,522 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-15 02:54:48,522 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-15 02:54:48,522 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-15 02:54:48,523 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:48,523 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:48,523 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-15 02:54:48,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:48,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:54:48,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:54:48,523 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:48,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:48,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:54:48,524 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:48,524 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-15 02:54:48,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-15 02:54:48,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:54:48,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:48,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:54:48,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:54:48,524 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:48,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:48,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:54:48,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:54:48,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:48,526 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:48,526 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:54:48,526 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:54:48,526 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:48,526 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-15 02:54:48,526 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-15 02:54:48,526 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-15 02:54:48,526 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-15 02:54:48,526 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:48,526 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:48,526 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:48,526 DEBUG [Thread-1527] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-15 02:54:48,526 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-15 02:54:48,526 DEBUG [Thread-1527] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-15 02:54:48,526 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-15 02:54:48,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-15 02:54:48,526 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-15 02:54:48,526 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-15 02:54:48,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-15 02:54:48,527 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f8160290180ba126c904acc32b60eac7, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:54:48,528 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34681] ipc.CallRunner(144): callId: 246 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:41918 deadline: 1681527348524, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f8160290180ba126c904acc32b60eac7, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:48,605 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=313 (bloomFilter=true), to=hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/f8160290180ba126c904acc32b60eac7/.tmp/A/4c132a56d87d4a4089c66bc0225f2389 2023-04-15 02:54:48,620 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=313 (bloomFilter=true), to=hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/f8160290180ba126c904acc32b60eac7/.tmp/B/9d7fee285c1d4831abc6577d7be84f9a 2023-04-15 02:54:48,626 DEBUG [Thread-1527] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-15 02:54:48,627 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-15 02:54:48,628 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-15 02:54:48,647 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-15 02:54:48,647 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-15 02:54:48,648 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-15 02:54:48,648 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-15 02:54:48,648 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-15 02:54:48,648 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-15 02:54:48,648 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:48,648 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=313 (bloomFilter=true), to=hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/f8160290180ba126c904acc32b60eac7/.tmp/C/cecdc7c51df8477f9e6f28d230be6ad0 2023-04-15 02:54:48,648 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:54:48,649 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-15 02:54:48,649 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:48,649 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-15 02:54:48,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-15 02:54:48,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:54:48,649 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:48,650 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:48,650 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-15 02:54:48,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-15 02:54:48,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:54:48,650 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-15 02:54:48,653 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/f8160290180ba126c904acc32b60eac7/.tmp/A/4c132a56d87d4a4089c66bc0225f2389 as hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/f8160290180ba126c904acc32b60eac7/A/4c132a56d87d4a4089c66bc0225f2389 2023-04-15 02:54:48,660 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/f8160290180ba126c904acc32b60eac7/A/4c132a56d87d4a4089c66bc0225f2389, entries=200, sequenceid=313, filesize=14.3 K 2023-04-15 02:54:48,661 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/f8160290180ba126c904acc32b60eac7/.tmp/B/9d7fee285c1d4831abc6577d7be84f9a as hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/f8160290180ba126c904acc32b60eac7/B/9d7fee285c1d4831abc6577d7be84f9a 2023-04-15 02:54:48,666 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-15 02:54:48,666 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-15 02:54:48,666 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-15 02:54:48,666 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-15 02:54:48,666 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-15 02:54:48,666 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-15 02:54:48,666 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-15 02:54:48,666 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681527274083.f8160290180ba126c904acc32b60eac7. 2023-04-15 02:54:48,666 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681527274083.f8160290180ba126c904acc32b60eac7. started... 2023-04-15 02:54:48,666 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681527274083.f8160290180ba126c904acc32b60eac7. as already flushing 2023-04-15 02:54:48,666 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681527274083.f8160290180ba126c904acc32b60eac7. 2023-04-15 02:54:48,667 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-15 02:54:48,667 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-15 02:54:48,667 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:48,667 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-15 02:54:48,667 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34681,1681527209133' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-15 02:54:48,667 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:48,667 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:48,668 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/f8160290180ba126c904acc32b60eac7/B/9d7fee285c1d4831abc6577d7be84f9a, entries=150, sequenceid=313, filesize=11.9 K 2023-04-15 02:54:48,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:48,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:54:48,668 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:48,668 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-15 02:54:48,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:54:48,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:54:48,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:54:48,668 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/f8160290180ba126c904acc32b60eac7/.tmp/C/cecdc7c51df8477f9e6f28d230be6ad0 as hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/f8160290180ba126c904acc32b60eac7/C/cecdc7c51df8477f9e6f28d230be6ad0 2023-04-15 02:54:48,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:54:48,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:48,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:54:48,669 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34681,1681527209133' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-15 02:54:48,669 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@208b1a61[Count = 0] remaining members to acquire global barrier 2023-04-15 02:54:48,669 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-15 02:54:48,669 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:48,670 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:48,670 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:48,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:48,670 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-15 02:54:48,670 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-15 02:54:48,670 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:48,670 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-15 02:54:48,670 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34681,1681527209133' in zk 2023-04-15 02:54:48,671 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-15 02:54:48,671 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:48,671 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-15 02:54:48,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:48,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:54:48,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:54:48,671 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-15 02:54:48,672 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/f8160290180ba126c904acc32b60eac7/C/cecdc7c51df8477f9e6f28d230be6ad0, entries=150, sequenceid=313, filesize=11.9 K 2023-04-15 02:54:48,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:54:48,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:54:48,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:54:48,672 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~140.89 KB/144270, heapSize ~405.56 KB/415296, currentSize=40.25 KB/41220 for f8160290180ba126c904acc32b60eac7 in 487ms, sequenceid=313, compaction requested=true 2023-04-15 02:54:48,672 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for f8160290180ba126c904acc32b60eac7: 2023-04-15 02:54:48,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:48,672 DEBUG [RS:0;jenkins-hbase20:34681-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 6 store files, 0 compacting, 6 eligible, 16 blocking 2023-04-15 02:54:48,672 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-15 02:54:48,672 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-15 02:54:48,672 DEBUG [RS:0;jenkins-hbase20:34681-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 6 store files, 0 compacting, 6 eligible, 16 blocking 2023-04-15 02:54:48,672 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-15 02:54:48,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:54:48,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:54:48,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:48,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34681,1681527209133': 2023-04-15 02:54:48,674 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34681,1681527209133' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-15 02:54:48,675 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-15 02:54:48,675 DEBUG [RS:0;jenkins-hbase20:34681-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 6 files of size 73054 starting at candidate #0 after considering 10 permutations with 10 in ratio 2023-04-15 02:54:48,675 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-15 02:54:48,675 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-15 02:54:48,675 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-15 02:54:48,675 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-15 02:54:48,675 DEBUG [RS:0;jenkins-hbase20:34681-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 6 files of size 70614 starting at candidate #0 after considering 10 permutations with 10 in ratio 2023-04-15 02:54:48,675 DEBUG [RS:0;jenkins-hbase20:34681-longCompactions-0] regionserver.HStore(1912): f8160290180ba126c904acc32b60eac7/B is initiating minor compaction (all files) 2023-04-15 02:54:48,675 DEBUG [RS:0;jenkins-hbase20:34681-shortCompactions-0] regionserver.HStore(1912): f8160290180ba126c904acc32b60eac7/A is initiating minor compaction (all files) 2023-04-15 02:54:48,675 INFO [RS:0;jenkins-hbase20:34681-longCompactions-0] regionserver.HRegion(2259): Starting compaction of f8160290180ba126c904acc32b60eac7/B in TestAcidGuarantees,,1681527274083.f8160290180ba126c904acc32b60eac7. 2023-04-15 02:54:48,675 INFO [RS:0;jenkins-hbase20:34681-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of f8160290180ba126c904acc32b60eac7/A in TestAcidGuarantees,,1681527274083.f8160290180ba126c904acc32b60eac7. 2023-04-15 02:54:48,675 INFO [RS:0;jenkins-hbase20:34681-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/f8160290180ba126c904acc32b60eac7/B/fffc50bcd93a4e7e9a049c6eddc9b678, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/f8160290180ba126c904acc32b60eac7/B/70be9726a32642b48c6b33242243860b, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/f8160290180ba126c904acc32b60eac7/B/e8e29ea90fdc4fdeb8353ff44c9293d2, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/f8160290180ba126c904acc32b60eac7/B/f035898cd29a4386814b629068363bef, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/f8160290180ba126c904acc32b60eac7/B/1af10683cc6a452aaece1b4f85a3c87b, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/f8160290180ba126c904acc32b60eac7/B/9d7fee285c1d4831abc6577d7be84f9a] into tmpdir=hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/f8160290180ba126c904acc32b60eac7/.tmp, totalSize=69.0 K 2023-04-15 02:54:48,675 INFO [RS:0;jenkins-hbase20:34681-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/f8160290180ba126c904acc32b60eac7/A/b8354584db9a4f43b0b4e9eef47e1892, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/f8160290180ba126c904acc32b60eac7/A/954aa6d0069848788bf78dade2b6f2e7, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/f8160290180ba126c904acc32b60eac7/A/6846a5a1400b41fb908682d63eb5d93e, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/f8160290180ba126c904acc32b60eac7/A/9276f313bec440ab82008fb21fe3271a, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/f8160290180ba126c904acc32b60eac7/A/774c69fb42ef48febac3611cacd2e05f, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/f8160290180ba126c904acc32b60eac7/A/4c132a56d87d4a4089c66bc0225f2389] into tmpdir=hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/f8160290180ba126c904acc32b60eac7/.tmp, totalSize=71.3 K 2023-04-15 02:54:48,676 DEBUG [RS:0;jenkins-hbase20:34681-longCompactions-0] compactions.Compactor(207): Compacting fffc50bcd93a4e7e9a049c6eddc9b678, keycount=150, bloomtype=ROW, size=12.1 K, encoding=NONE, compression=NONE, seqNum=178, earliestPutTs=1681527283131 2023-04-15 02:54:48,678 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:48,678 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:48,678 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-15 02:54:48,678 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:48,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:48,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:48,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:54:48,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:54:48,679 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-15 02:54:48,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-15 02:54:48,679 DEBUG [RS:0;jenkins-hbase20:34681-longCompactions-0] compactions.Compactor(207): Compacting 70be9726a32642b48c6b33242243860b, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=215, earliestPutTs=1681527283259 2023-04-15 02:54:48,679 DEBUG [RS:0;jenkins-hbase20:34681-shortCompactions-0] compactions.Compactor(207): Compacting b8354584db9a4f43b0b4e9eef47e1892, keycount=150, bloomtype=ROW, size=12.1 K, encoding=NONE, compression=NONE, seqNum=178, earliestPutTs=1681527283131 2023-04-15 02:54:48,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:48,679 DEBUG [RS:0;jenkins-hbase20:34681-longCompactions-0] compactions.Compactor(207): Compacting e8e29ea90fdc4fdeb8353ff44c9293d2, keycount=100, bloomtype=ROW, size=9.4 K, encoding=NONE, compression=NONE, seqNum=228, earliestPutTs=1681527283919 2023-04-15 02:54:48,679 DEBUG [RS:0;jenkins-hbase20:34681-shortCompactions-0] compactions.Compactor(207): Compacting 954aa6d0069848788bf78dade2b6f2e7, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=215, earliestPutTs=1681527283259 2023-04-15 02:54:48,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:54:48,682 DEBUG [RS:0;jenkins-hbase20:34681-longCompactions-0] compactions.Compactor(207): Compacting f035898cd29a4386814b629068363bef, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=265, earliestPutTs=1681527285150 2023-04-15 02:54:48,683 DEBUG [RS:0;jenkins-hbase20:34681-shortCompactions-0] compactions.Compactor(207): Compacting 6846a5a1400b41fb908682d63eb5d93e, keycount=100, bloomtype=ROW, size=9.4 K, encoding=NONE, compression=NONE, seqNum=228, earliestPutTs=1681527283919 2023-04-15 02:54:48,683 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:48,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:54:48,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:54:48,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:54:48,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:48,691 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:48,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:54:48,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:54:48,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:48,693 DEBUG [RS:0;jenkins-hbase20:34681-longCompactions-0] compactions.Compactor(207): Compacting 1af10683cc6a452aaece1b4f85a3c87b, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=275, earliestPutTs=1681527286276 2023-04-15 02:54:48,693 DEBUG [RS:0;jenkins-hbase20:34681-shortCompactions-0] compactions.Compactor(207): Compacting 9276f313bec440ab82008fb21fe3271a, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=265, earliestPutTs=1681527285150 2023-04-15 02:54:48,693 DEBUG [RS:0;jenkins-hbase20:34681-longCompactions-0] compactions.Compactor(207): Compacting 9d7fee285c1d4831abc6577d7be84f9a, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=313, earliestPutTs=1681527287039 2023-04-15 02:54:48,693 DEBUG [RS:0;jenkins-hbase20:34681-shortCompactions-0] compactions.Compactor(207): Compacting 774c69fb42ef48febac3611cacd2e05f, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=275, earliestPutTs=1681527286276 2023-04-15 02:54:48,694 DEBUG [RS:0;jenkins-hbase20:34681-shortCompactions-0] compactions.Compactor(207): Compacting 4c132a56d87d4a4089c66bc0225f2389, keycount=200, bloomtype=ROW, size=14.3 K, encoding=NONE, compression=NONE, seqNum=313, earliestPutTs=1681527287039 2023-04-15 02:54:48,694 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-15 02:54:48,694 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-15 02:54:48,694 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-15 02:54:48,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-15 02:54:48,695 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:48,695 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:54:48,695 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-15 02:54:48,695 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-15 02:54:48,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-15 02:54:48,695 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:54:48,695 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:48,695 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:48,695 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:48,695 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:48,695 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-15 02:54:48,695 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-15 02:54:48,696 DEBUG [Thread-1527] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-15 02:54:48,696 DEBUG [Thread-1527] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-15 02:54:48,713 INFO [RS:0;jenkins-hbase20:34681-longCompactions-0] throttle.PressureAwareThroughputController(145): f8160290180ba126c904acc32b60eac7#B#compaction#227 average throughput is 2.18 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-15 02:54:48,716 INFO [RS:0;jenkins-hbase20:34681-shortCompactions-0] throttle.PressureAwareThroughputController(145): f8160290180ba126c904acc32b60eac7#A#compaction#228 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-15 02:54:48,796 DEBUG [Thread-1527] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-15 02:54:48,796 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-15 02:54:48,798 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-15 02:54:48,806 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-15 02:54:48,806 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-15 02:54:48,806 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-15 02:54:48,806 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-15 02:54:48,806 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-15 02:54:48,806 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-15 02:54:48,807 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:48,807 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:54:48,808 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:48,808 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-15 02:54:48,808 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-15 02:54:48,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-15 02:54:48,808 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:48,808 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-15 02:54:48,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:54:48,808 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:48,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-15 02:54:48,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:54:48,809 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-15 02:54:48,818 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-15 02:54:48,818 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-15 02:54:48,818 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-15 02:54:48,818 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-15 02:54:48,818 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-15 02:54:48,818 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-15 02:54:48,818 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-15 02:54:48,818 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681527274083.f8160290180ba126c904acc32b60eac7. 2023-04-15 02:54:48,818 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681527274083.f8160290180ba126c904acc32b60eac7. started... 2023-04-15 02:54:48,818 INFO [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing f8160290180ba126c904acc32b60eac7 3/3 column families, dataSize=40.25 KB heapSize=120.28 KB 2023-04-15 02:54:48,819 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK f8160290180ba126c904acc32b60eac7, store=A 2023-04-15 02:54:48,819 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-15 02:54:48,819 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK f8160290180ba126c904acc32b60eac7, store=B 2023-04-15 02:54:48,819 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-15 02:54:48,819 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK f8160290180ba126c904acc32b60eac7, store=C 2023-04-15 02:54:48,819 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-15 02:54:48,831 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34681] regionserver.HRegion(9158): Flush requested on f8160290180ba126c904acc32b60eac7 2023-04-15 02:54:48,831 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681527274083.f8160290180ba126c904acc32b60eac7. as already flushing 2023-04-15 02:54:48,898 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681527288884, max timestamp=1681527288897Num uniques 859; 2023-04-15 02:54:48,899 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681527288884, max timestamp=1681527288897Num uniques 859; 2023-04-15 02:54:48,899 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681527288884, max timestamp=1681527288897Num uniques 859; 2023-04-15 02:54:48,899 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-1] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-15 02:54:48,899 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-6] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-15 02:54:48,899 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-7] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-15 02:54:48,985 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=168, cellSize=7688, totalHeapSize=17712, min timestamp=1681527288963, max timestamp=1681527288983Num uniques 1018; 2023-04-15 02:54:48,985 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-8] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=7688, suffix heap size=133840,new segment heap size=17712 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=168 2023-04-15 02:54:48,986 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=168, cellSize=7688, totalHeapSize=17712, min timestamp=1681527288963, max timestamp=1681527288983Num uniques 1018; 2023-04-15 02:54:48,986 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=168, cellSize=7688, totalHeapSize=17712, min timestamp=1681527288963, max timestamp=1681527288983Num uniques 1018; 2023-04-15 02:54:48,986 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=7688, suffix heap size=133840,new segment heap size=17712 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=168 2023-04-15 02:54:48,986 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-2] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=7688, suffix heap size=133840,new segment heap size=17712 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=168 2023-04-15 02:54:49,064 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=177, cellSize=8102, totalHeapSize=18648, min timestamp=1681527289046, max timestamp=1681527289061Num uniques 1027; 2023-04-15 02:54:49,064 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=177, cellSize=8102, totalHeapSize=18648, min timestamp=1681527289046, max timestamp=1681527289061Num uniques 1027; 2023-04-15 02:54:49,064 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-9] regionserver.CompactionPipeline(163): Suffix data size=47032, new segment data size=8102, suffix heap size=134776,new segment heap size=18648 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1027, new segment cells count=177 2023-04-15 02:54:49,064 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-3] regionserver.CompactionPipeline(163): Suffix data size=47032, new segment data size=8102, suffix heap size=134776,new segment heap size=18648 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1027, new segment cells count=177 2023-04-15 02:54:49,064 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=177, cellSize=8102, totalHeapSize=18648, min timestamp=1681527289046, max timestamp=1681527289061Num uniques 1027; 2023-04-15 02:54:49,064 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-4] regionserver.CompactionPipeline(163): Suffix data size=47032, new segment data size=8102, suffix heap size=134776,new segment heap size=18648 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1027, new segment cells count=177 2023-04-15 02:54:49,122 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=186, cellSize=8516, totalHeapSize=19584, min timestamp=1681527289099, max timestamp=1681527289120Num uniques 1036; 2023-04-15 02:54:49,122 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-6] regionserver.CompactionPipeline(163): Suffix data size=47446, new segment data size=8516, suffix heap size=135712,new segment heap size=19584 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1036, new segment cells count=186 2023-04-15 02:54:49,122 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=186, cellSize=8516, totalHeapSize=19584, min timestamp=1681527289099, max timestamp=1681527289120Num uniques 1036; 2023-04-15 02:54:49,122 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-5] regionserver.CompactionPipeline(163): Suffix data size=47446, new segment data size=8516, suffix heap size=135712,new segment heap size=19584 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1036, new segment cells count=186 2023-04-15 02:54:49,122 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=186, cellSize=8516, totalHeapSize=19584, min timestamp=1681527289099, max timestamp=1681527289120Num uniques 1036; 2023-04-15 02:54:49,123 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-1] regionserver.CompactionPipeline(163): Suffix data size=47446, new segment data size=8516, suffix heap size=135712,new segment heap size=19584 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1036, new segment cells count=186 2023-04-15 02:54:49,151 DEBUG [RS:0;jenkins-hbase20:34681-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/f8160290180ba126c904acc32b60eac7/.tmp/A/e73d4d3b96aa40cd914bc600fa1a7979 as hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/f8160290180ba126c904acc32b60eac7/A/e73d4d3b96aa40cd914bc600fa1a7979 2023-04-15 02:54:49,151 DEBUG [RS:0;jenkins-hbase20:34681-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/f8160290180ba126c904acc32b60eac7/.tmp/B/4eca2f71b8374fbd86068b6b3f1136e8 as hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/f8160290180ba126c904acc32b60eac7/B/4eca2f71b8374fbd86068b6b3f1136e8 2023-04-15 02:54:49,156 INFO [RS:0;jenkins-hbase20:34681-longCompactions-0] regionserver.HStore(1652): Completed compaction of 6 (all) file(s) in f8160290180ba126c904acc32b60eac7/B of f8160290180ba126c904acc32b60eac7 into 4eca2f71b8374fbd86068b6b3f1136e8(size=12.4 K), total size for store is 12.4 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-15 02:54:49,156 DEBUG [RS:0;jenkins-hbase20:34681-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for f8160290180ba126c904acc32b60eac7: 2023-04-15 02:54:49,156 INFO [RS:0;jenkins-hbase20:34681-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681527274083.f8160290180ba126c904acc32b60eac7., storeName=f8160290180ba126c904acc32b60eac7/B, priority=10, startTime=1681527288672; duration=0sec 2023-04-15 02:54:49,156 DEBUG [RS:0;jenkins-hbase20:34681-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-15 02:54:49,156 DEBUG [RS:0;jenkins-hbase20:34681-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 6 store files, 0 compacting, 6 eligible, 16 blocking 2023-04-15 02:54:49,157 INFO [RS:0;jenkins-hbase20:34681-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 6 (all) file(s) in f8160290180ba126c904acc32b60eac7/A of f8160290180ba126c904acc32b60eac7 into e73d4d3b96aa40cd914bc600fa1a7979(size=12.4 K), total size for store is 12.4 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-15 02:54:49,157 DEBUG [RS:0;jenkins-hbase20:34681-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for f8160290180ba126c904acc32b60eac7: 2023-04-15 02:54:49,157 INFO [RS:0;jenkins-hbase20:34681-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681527274083.f8160290180ba126c904acc32b60eac7., storeName=f8160290180ba126c904acc32b60eac7/A, priority=10, startTime=1681527288672; duration=0sec 2023-04-15 02:54:49,157 DEBUG [RS:0;jenkins-hbase20:34681-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-15 02:54:49,158 DEBUG [RS:0;jenkins-hbase20:34681-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 6 files of size 70614 starting at candidate #0 after considering 10 permutations with 10 in ratio 2023-04-15 02:54:49,158 DEBUG [RS:0;jenkins-hbase20:34681-longCompactions-0] regionserver.HStore(1912): f8160290180ba126c904acc32b60eac7/C is initiating minor compaction (all files) 2023-04-15 02:54:49,158 INFO [RS:0;jenkins-hbase20:34681-longCompactions-0] regionserver.HRegion(2259): Starting compaction of f8160290180ba126c904acc32b60eac7/C in TestAcidGuarantees,,1681527274083.f8160290180ba126c904acc32b60eac7. 2023-04-15 02:54:49,158 INFO [RS:0;jenkins-hbase20:34681-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/f8160290180ba126c904acc32b60eac7/C/041c340efc9c4b908be2bec6a35b7f72, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/f8160290180ba126c904acc32b60eac7/C/01c51da9cec44adcb344482e733bcb14, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/f8160290180ba126c904acc32b60eac7/C/3a07d2aaddaa4447a6531f316856f1cc, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/f8160290180ba126c904acc32b60eac7/C/9b2876c82e6d44f0b51ff6f305538d23, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/f8160290180ba126c904acc32b60eac7/C/7807ee8f8b164f238aca4f3363e0ee36, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/f8160290180ba126c904acc32b60eac7/C/cecdc7c51df8477f9e6f28d230be6ad0] into tmpdir=hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/f8160290180ba126c904acc32b60eac7/.tmp, totalSize=69.0 K 2023-04-15 02:54:49,159 DEBUG [RS:0;jenkins-hbase20:34681-longCompactions-0] compactions.Compactor(207): Compacting 041c340efc9c4b908be2bec6a35b7f72, keycount=150, bloomtype=ROW, size=12.1 K, encoding=NONE, compression=NONE, seqNum=178, earliestPutTs=1681527283131 2023-04-15 02:54:49,159 DEBUG [RS:0;jenkins-hbase20:34681-longCompactions-0] compactions.Compactor(207): Compacting 01c51da9cec44adcb344482e733bcb14, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=215, earliestPutTs=1681527283259 2023-04-15 02:54:49,160 DEBUG [RS:0;jenkins-hbase20:34681-longCompactions-0] compactions.Compactor(207): Compacting 3a07d2aaddaa4447a6531f316856f1cc, keycount=100, bloomtype=ROW, size=9.4 K, encoding=NONE, compression=NONE, seqNum=228, earliestPutTs=1681527283919 2023-04-15 02:54:49,160 DEBUG [RS:0;jenkins-hbase20:34681-longCompactions-0] compactions.Compactor(207): Compacting 9b2876c82e6d44f0b51ff6f305538d23, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=265, earliestPutTs=1681527285150 2023-04-15 02:54:49,160 DEBUG [RS:0;jenkins-hbase20:34681-longCompactions-0] compactions.Compactor(207): Compacting 7807ee8f8b164f238aca4f3363e0ee36, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=275, earliestPutTs=1681527286276 2023-04-15 02:54:49,160 DEBUG [RS:0;jenkins-hbase20:34681-longCompactions-0] compactions.Compactor(207): Compacting cecdc7c51df8477f9e6f28d230be6ad0, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=313, earliestPutTs=1681527287039 2023-04-15 02:54:49,169 INFO [RS:0;jenkins-hbase20:34681-longCompactions-0] throttle.PressureAwareThroughputController(145): f8160290180ba126c904acc32b60eac7#C#compaction#230 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-15 02:54:49,186 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=195, cellSize=8930, totalHeapSize=20520, min timestamp=1681527289140, max timestamp=1681527289183Num uniques 1045; 2023-04-15 02:54:49,186 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-7] regionserver.CompactionPipeline(163): Suffix data size=47860, new segment data size=8930, suffix heap size=136648,new segment heap size=20520 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1045, new segment cells count=195 2023-04-15 02:54:49,186 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=195, cellSize=8930, totalHeapSize=20520, min timestamp=1681527289140, max timestamp=1681527289183Num uniques 1045; 2023-04-15 02:54:49,186 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-8] regionserver.CompactionPipeline(163): Suffix data size=47860, new segment data size=8930, suffix heap size=136648,new segment heap size=20520 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1045, new segment cells count=195 2023-04-15 02:54:49,187 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=195, cellSize=8930, totalHeapSize=20520, min timestamp=1681527289140, max timestamp=1681527289183Num uniques 1045; 2023-04-15 02:54:49,187 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(163): Suffix data size=47860, new segment data size=8930, suffix heap size=136648,new segment heap size=20520 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1045, new segment cells count=195 2023-04-15 02:54:49,239 INFO [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=13.42 KB at sequenceid=322 (bloomFilter=true), to=hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/f8160290180ba126c904acc32b60eac7/.tmp/A/0c0af3ff076a46db963ed21c3b7e3d42 2023-04-15 02:54:49,251 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f8160290180ba126c904acc32b60eac7, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:54:49,251 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34681] ipc.CallRunner(144): callId: 351 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:41918 deadline: 1681527349248, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f8160290180ba126c904acc32b60eac7, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:49,259 INFO [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=13.42 KB at sequenceid=322 (bloomFilter=true), to=hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/f8160290180ba126c904acc32b60eac7/.tmp/B/48c53c106b4642519b29b1e401289ef7 2023-04-15 02:54:49,293 INFO [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=13.42 KB at sequenceid=322 (bloomFilter=true), to=hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/f8160290180ba126c904acc32b60eac7/.tmp/C/a6113b6a91f3426c92e0b038b548780a 2023-04-15 02:54:49,297 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/f8160290180ba126c904acc32b60eac7/.tmp/A/0c0af3ff076a46db963ed21c3b7e3d42 as hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/f8160290180ba126c904acc32b60eac7/A/0c0af3ff076a46db963ed21c3b7e3d42 2023-04-15 02:54:49,301 INFO [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/f8160290180ba126c904acc32b60eac7/A/0c0af3ff076a46db963ed21c3b7e3d42, entries=100, sequenceid=322, filesize=9.5 K 2023-04-15 02:54:49,301 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/f8160290180ba126c904acc32b60eac7/.tmp/B/48c53c106b4642519b29b1e401289ef7 as hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/f8160290180ba126c904acc32b60eac7/B/48c53c106b4642519b29b1e401289ef7 2023-04-15 02:54:49,305 INFO [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/f8160290180ba126c904acc32b60eac7/B/48c53c106b4642519b29b1e401289ef7, entries=100, sequenceid=322, filesize=9.5 K 2023-04-15 02:54:49,306 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/f8160290180ba126c904acc32b60eac7/.tmp/C/a6113b6a91f3426c92e0b038b548780a as hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/f8160290180ba126c904acc32b60eac7/C/a6113b6a91f3426c92e0b038b548780a 2023-04-15 02:54:49,312 INFO [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/f8160290180ba126c904acc32b60eac7/C/a6113b6a91f3426c92e0b038b548780a, entries=100, sequenceid=322, filesize=9.5 K 2023-04-15 02:54:49,313 INFO [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~40.25 KB/41220, heapSize ~120.23 KB/123120, currentSize=140.89 KB/144270 for f8160290180ba126c904acc32b60eac7 in 495ms, sequenceid=322, compaction requested=false 2023-04-15 02:54:49,313 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for f8160290180ba126c904acc32b60eac7: 2023-04-15 02:54:49,313 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681527274083.f8160290180ba126c904acc32b60eac7. 2023-04-15 02:54:49,313 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-15 02:54:49,313 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-15 02:54:49,313 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:49,314 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-15 02:54:49,314 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34681,1681527209133' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-15 02:54:49,315 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:49,315 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:49,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:49,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:54:49,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:54:49,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:54:49,315 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:49,315 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-15 02:54:49,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:54:49,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:54:49,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:49,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:54:49,317 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34681,1681527209133' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-15 02:54:49,317 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5f0d50bc[Count = 0] remaining members to acquire global barrier 2023-04-15 02:54:49,317 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-15 02:54:49,317 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:49,318 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:49,318 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:49,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:49,318 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-15 02:54:49,318 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-15 02:54:49,318 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34681,1681527209133' in zk 2023-04-15 02:54:49,318 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:49,318 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-15 02:54:49,319 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-15 02:54:49,319 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-15 02:54:49,319 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-15 02:54:49,319 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:49,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:49,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:54:49,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:54:49,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:54:49,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:54:49,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:54:49,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:49,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:54:49,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:54:49,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:49,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34681,1681527209133': 2023-04-15 02:54:49,325 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34681,1681527209133' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-15 02:54:49,325 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-15 02:54:49,325 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-15 02:54:49,325 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-15 02:54:49,325 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-15 02:54:49,325 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-15 02:54:49,332 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:49,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:49,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:54:49,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:54:49,333 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:49,333 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-15 02:54:49,333 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:49,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:49,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:54:49,333 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-15 02:54:49,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-15 02:54:49,333 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:49,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:54:49,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:49,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:54:49,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:54:49,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:49,335 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:49,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:54:49,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:54:49,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:49,336 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-15 02:54:49,336 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-15 02:54:49,336 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:49,336 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-15 02:54:49,337 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-15 02:54:49,337 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-15 02:54:49,337 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-15 02:54:49,337 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-15 02:54:49,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-15 02:54:49,337 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:54:49,337 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:54:49,337 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:49,337 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:49,337 DEBUG [Thread-1527] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-15 02:54:49,337 DEBUG [Thread-1527] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-15 02:54:49,337 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-15 02:54:49,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-15 02:54:49,337 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:49,337 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:49,354 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34681] regionserver.HRegion(9158): Flush requested on f8160290180ba126c904acc32b60eac7 2023-04-15 02:54:49,354 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing f8160290180ba126c904acc32b60eac7 3/3 column families, dataSize=147.60 KB heapSize=422.16 KB 2023-04-15 02:54:49,355 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK f8160290180ba126c904acc32b60eac7, store=A 2023-04-15 02:54:49,355 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=3, new segment=null 2023-04-15 02:54:49,355 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK f8160290180ba126c904acc32b60eac7, store=B 2023-04-15 02:54:49,355 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=3, new segment=null 2023-04-15 02:54:49,355 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK f8160290180ba126c904acc32b60eac7, store=C 2023-04-15 02:54:49,355 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=3, new segment=null 2023-04-15 02:54:49,380 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f8160290180ba126c904acc32b60eac7, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:54:49,380 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34681] ipc.CallRunner(144): callId: 359 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:41918 deadline: 1681527349375, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f8160290180ba126c904acc32b60eac7, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:49,437 DEBUG [Thread-1527] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-15 02:54:49,438 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-15 02:54:49,439 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-15 02:54:49,439 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-15 02:54:49,440 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-15 02:54:49,440 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-15 02:54:49,440 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-15 02:54:49,440 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-15 02:54:49,440 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-15 02:54:49,440 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:49,440 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:54:49,441 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-15 02:54:49,441 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:49,441 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-15 02:54:49,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-15 02:54:49,441 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:49,441 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-15 02:54:49,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:54:49,441 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:49,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-15 02:54:49,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:54:49,442 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-15 02:54:49,442 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-15 02:54:49,442 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-15 02:54:49,442 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-15 02:54:49,442 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-15 02:54:49,442 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-15 02:54:49,446 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-15 02:54:49,446 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-15 02:54:49,446 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681527274083.f8160290180ba126c904acc32b60eac7. 2023-04-15 02:54:49,446 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681527274083.f8160290180ba126c904acc32b60eac7. started... 2023-04-15 02:54:49,446 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681527274083.f8160290180ba126c904acc32b60eac7. as already flushing 2023-04-15 02:54:49,446 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681527274083.f8160290180ba126c904acc32b60eac7. 2023-04-15 02:54:49,446 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-15 02:54:49,446 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-15 02:54:49,446 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:49,446 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-15 02:54:49,446 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34681,1681527209133' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-15 02:54:49,447 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:49,447 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:49,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:49,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:54:49,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:54:49,447 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:49,447 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-15 02:54:49,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:54:49,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:54:49,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:54:49,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:49,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:54:49,448 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34681,1681527209133' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-15 02:54:49,449 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2660ac33[Count = 0] remaining members to acquire global barrier 2023-04-15 02:54:49,449 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-15 02:54:49,449 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:49,449 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:49,449 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:49,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:49,449 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-15 02:54:49,449 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-15 02:54:49,449 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34681,1681527209133' in zk 2023-04-15 02:54:49,449 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:49,449 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-15 02:54:49,450 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-15 02:54:49,450 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-15 02:54:49,450 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:49,450 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-15 02:54:49,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:49,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:54:49,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:54:49,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:54:49,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:54:49,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:54:49,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:49,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:54:49,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:54:49,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:49,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34681,1681527209133': 2023-04-15 02:54:49,452 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34681,1681527209133' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-15 02:54:49,452 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-15 02:54:49,452 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-15 02:54:49,452 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-15 02:54:49,452 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-15 02:54:49,452 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-15 02:54:49,453 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:49,453 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:49,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:49,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:54:49,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:54:49,453 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:49,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:49,453 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-15 02:54:49,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:54:49,453 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-15 02:54:49,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-15 02:54:49,453 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:49,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:54:49,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:49,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:54:49,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:54:49,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:49,454 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:49,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:54:49,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:54:49,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:49,455 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:49,456 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-15 02:54:49,456 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:54:49,456 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-15 02:54:49,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-15 02:54:49,456 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-15 02:54:49,456 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-15 02:54:49,456 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-15 02:54:49,456 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:54:49,456 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-15 02:54:49,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-15 02:54:49,456 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-15 02:54:49,456 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:49,456 DEBUG [Thread-1527] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-15 02:54:49,456 DEBUG [Thread-1527] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-15 02:54:49,456 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:49,456 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:49,456 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:49,484 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f8160290180ba126c904acc32b60eac7, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:54:49,485 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34681] ipc.CallRunner(144): callId: 361 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:41918 deadline: 1681527349481, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f8160290180ba126c904acc32b60eac7, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:49,556 DEBUG [Thread-1527] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-15 02:54:49,557 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-15 02:54:49,562 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-15 02:54:49,564 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-15 02:54:49,564 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-15 02:54:49,564 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-15 02:54:49,564 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-15 02:54:49,564 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-15 02:54:49,564 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-15 02:54:49,564 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:49,564 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:54:49,565 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:49,565 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-15 02:54:49,565 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-15 02:54:49,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-15 02:54:49,565 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:49,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:54:49,565 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-15 02:54:49,566 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:49,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-15 02:54:49,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:54:49,566 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-15 02:54:49,566 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-15 02:54:49,566 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-15 02:54:49,566 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-15 02:54:49,566 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-15 02:54:49,566 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-15 02:54:49,566 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-15 02:54:49,567 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-15 02:54:49,567 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681527274083.f8160290180ba126c904acc32b60eac7. 2023-04-15 02:54:49,567 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681527274083.f8160290180ba126c904acc32b60eac7. started... 2023-04-15 02:54:49,567 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681527274083.f8160290180ba126c904acc32b60eac7. as already flushing 2023-04-15 02:54:49,567 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681527274083.f8160290180ba126c904acc32b60eac7. 2023-04-15 02:54:49,567 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-15 02:54:49,567 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-15 02:54:49,567 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:49,567 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-15 02:54:49,567 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34681,1681527209133' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-15 02:54:49,568 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:49,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:49,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:54:49,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:54:49,568 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:49,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:54:49,568 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:49,568 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-15 02:54:49,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:54:49,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:54:49,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:49,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:54:49,569 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34681,1681527209133' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-15 02:54:49,569 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@70f0e14c[Count = 0] remaining members to acquire global barrier 2023-04-15 02:54:49,569 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-15 02:54:49,569 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:49,571 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:49,571 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:49,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:49,571 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-15 02:54:49,571 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-15 02:54:49,571 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:49,571 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-15 02:54:49,571 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34681,1681527209133' in zk 2023-04-15 02:54:49,572 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:49,572 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-15 02:54:49,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:49,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:54:49,572 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-15 02:54:49,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:54:49,572 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-15 02:54:49,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:54:49,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:54:49,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:54:49,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:49,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:54:49,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:54:49,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:49,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34681,1681527209133': 2023-04-15 02:54:49,574 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34681,1681527209133' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-15 02:54:49,574 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-15 02:54:49,574 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-15 02:54:49,574 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-15 02:54:49,574 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-15 02:54:49,574 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-15 02:54:49,575 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:49,575 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-15 02:54:49,575 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:49,575 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:49,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:49,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:54:49,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:54:49,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:49,575 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:49,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:54:49,575 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-15 02:54:49,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-15 02:54:49,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:54:49,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:49,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:54:49,576 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:49,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:54:49,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:49,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:54:49,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:54:49,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:49,579 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:49,579 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-15 02:54:49,579 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:54:49,579 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:54:49,579 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:49,579 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-15 02:54:49,579 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-15 02:54:49,579 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-15 02:54:49,579 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-15 02:54:49,579 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-15 02:54:49,579 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-15 02:54:49,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-15 02:54:49,580 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-15 02:54:49,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-15 02:54:49,579 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:49,580 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:49,580 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:49,579 DEBUG [Thread-1527] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-15 02:54:49,580 DEBUG [Thread-1527] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-15 02:54:49,588 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f8160290180ba126c904acc32b60eac7, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:54:49,588 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34681] ipc.CallRunner(144): callId: 67 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41916 deadline: 1681527349585, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f8160290180ba126c904acc32b60eac7, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:49,589 DEBUG [Thread-1523] client.RpcRetryingCallerImpl(129): Call exception, tries=7, retries=16, started=8170 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f8160290180ba126c904acc32b60eac7, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_1' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1681527274083.f8160290180ba126c904acc32b60eac7., hostname=jenkins-hbase20.apache.org,34681,1681527209133, seqNum=2, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f8160290180ba126c904acc32b60eac7, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor31.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f8160290180ba126c904acc32b60eac7, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-15 02:54:49,592 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f8160290180ba126c904acc32b60eac7, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:54:49,592 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34681] ipc.CallRunner(144): callId: 66 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41940 deadline: 1681527349590, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f8160290180ba126c904acc32b60eac7, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:49,595 DEBUG [Thread-1521] client.RpcRetryingCallerImpl(129): Call exception, tries=7, retries=16, started=8180 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f8160290180ba126c904acc32b60eac7, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_2' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1681527274083.f8160290180ba126c904acc32b60eac7., hostname=jenkins-hbase20.apache.org,34681,1681527209133, seqNum=2, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f8160290180ba126c904acc32b60eac7, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor31.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f8160290180ba126c904acc32b60eac7, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-15 02:54:49,599 DEBUG [RS:0;jenkins-hbase20:34681-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/f8160290180ba126c904acc32b60eac7/.tmp/C/378b0b8eb7bd42889792cec86a396fda as hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/f8160290180ba126c904acc32b60eac7/C/378b0b8eb7bd42889792cec86a396fda 2023-04-15 02:54:49,609 INFO [RS:0;jenkins-hbase20:34681-longCompactions-0] regionserver.HStore(1652): Completed compaction of 6 (all) file(s) in f8160290180ba126c904acc32b60eac7/C of f8160290180ba126c904acc32b60eac7 into 378b0b8eb7bd42889792cec86a396fda(size=12.4 K), total size for store is 21.9 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-15 02:54:49,609 DEBUG [RS:0;jenkins-hbase20:34681-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for f8160290180ba126c904acc32b60eac7: 2023-04-15 02:54:49,609 INFO [RS:0;jenkins-hbase20:34681-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681527274083.f8160290180ba126c904acc32b60eac7., storeName=f8160290180ba126c904acc32b60eac7/C, priority=10, startTime=1681527288672; duration=0sec 2023-04-15 02:54:49,610 DEBUG [RS:0;jenkins-hbase20:34681-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-15 02:54:49,618 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f8160290180ba126c904acc32b60eac7, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:54:49,618 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34681] ipc.CallRunner(144): callId: 73 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41890 deadline: 1681527349611, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f8160290180ba126c904acc32b60eac7, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:49,620 DEBUG [Thread-1517] client.RpcRetryingCallerImpl(129): Call exception, tries=7, retries=16, started=8201 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f8160290180ba126c904acc32b60eac7, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_0' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1681527274083.f8160290180ba126c904acc32b60eac7., hostname=jenkins-hbase20.apache.org,34681,1681527209133, seqNum=2, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f8160290180ba126c904acc32b60eac7, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor31.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f8160290180ba126c904acc32b60eac7, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-15 02:54:49,623 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f8160290180ba126c904acc32b60eac7, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:54:49,623 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34681] ipc.CallRunner(144): callId: 77 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41926 deadline: 1681527349618, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f8160290180ba126c904acc32b60eac7, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:49,624 DEBUG [Thread-1525] client.RpcRetryingCallerImpl(129): Call exception, tries=7, retries=16, started=8205 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f8160290180ba126c904acc32b60eac7, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_1' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1681527274083.f8160290180ba126c904acc32b60eac7., hostname=jenkins-hbase20.apache.org,34681,1681527209133, seqNum=2, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f8160290180ba126c904acc32b60eac7, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor31.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f8160290180ba126c904acc32b60eac7, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-15 02:54:49,680 DEBUG [Thread-1527] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-15 02:54:49,681 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-15 02:54:49,682 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-15 02:54:49,683 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-15 02:54:49,683 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-15 02:54:49,683 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-15 02:54:49,683 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-15 02:54:49,683 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-15 02:54:49,683 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-15 02:54:49,683 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:49,683 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:54:49,684 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-15 02:54:49,684 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:49,684 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-15 02:54:49,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-15 02:54:49,684 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:49,684 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-15 02:54:49,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:54:49,685 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:49,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-15 02:54:49,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:54:49,685 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-15 02:54:49,685 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-15 02:54:49,685 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-15 02:54:49,685 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-15 02:54:49,685 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-15 02:54:49,685 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-15 02:54:49,685 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-15 02:54:49,685 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-15 02:54:49,685 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681527274083.f8160290180ba126c904acc32b60eac7. 2023-04-15 02:54:49,685 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681527274083.f8160290180ba126c904acc32b60eac7. started... 2023-04-15 02:54:49,685 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681527274083.f8160290180ba126c904acc32b60eac7. as already flushing 2023-04-15 02:54:49,686 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681527274083.f8160290180ba126c904acc32b60eac7. 2023-04-15 02:54:49,686 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-15 02:54:49,686 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-15 02:54:49,686 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:49,686 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-15 02:54:49,686 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34681,1681527209133' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-15 02:54:49,686 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:49,686 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:49,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:49,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:54:49,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:54:49,686 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f8160290180ba126c904acc32b60eac7, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:54:49,687 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34681] ipc.CallRunner(144): callId: 363 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:41918 deadline: 1681527349686, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f8160290180ba126c904acc32b60eac7, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:49,687 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:49,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:54:49,687 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-15 02:54:49,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:54:49,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:54:49,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:49,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:54:49,688 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34681,1681527209133' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-15 02:54:49,688 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@8d0b521[Count = 0] remaining members to acquire global barrier 2023-04-15 02:54:49,688 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-15 02:54:49,688 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:49,688 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:49,688 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:49,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:49,688 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-15 02:54:49,688 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-15 02:54:49,688 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34681,1681527209133' in zk 2023-04-15 02:54:49,688 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:49,689 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-15 02:54:49,689 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:49,689 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-15 02:54:49,689 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-15 02:54:49,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:49,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:54:49,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:54:49,689 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-15 02:54:49,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:54:49,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:54:49,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:54:49,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:49,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:54:49,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:54:49,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:49,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34681,1681527209133': 2023-04-15 02:54:49,691 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34681,1681527209133' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-15 02:54:49,691 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-15 02:54:49,691 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-15 02:54:49,691 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-15 02:54:49,691 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-15 02:54:49,691 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-15 02:54:49,692 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:49,692 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-15 02:54:49,692 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:49,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:49,692 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:49,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:49,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:54:49,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:54:49,692 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:49,692 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-15 02:54:49,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-15 02:54:49,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:54:49,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:49,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:54:49,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:54:49,693 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:49,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:54:49,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:49,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:54:49,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:54:49,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:49,694 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:49,694 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:54:49,694 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:54:49,694 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-15 02:54:49,694 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-15 02:54:49,694 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-15 02:54:49,694 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:49,694 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-15 02:54:49,695 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:49,694 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-15 02:54:49,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-15 02:54:49,694 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-15 02:54:49,695 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:49,695 DEBUG [Thread-1527] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-15 02:54:49,695 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:49,695 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-15 02:54:49,695 DEBUG [Thread-1527] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-15 02:54:49,695 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-15 02:54:49,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-15 02:54:49,763 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=49.20 KB at sequenceid=431 (bloomFilter=true), to=hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/f8160290180ba126c904acc32b60eac7/.tmp/A/8ec77327a35847be8ff2bba511070406 2023-04-15 02:54:49,795 DEBUG [Thread-1527] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-15 02:54:49,796 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-15 02:54:49,797 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-15 02:54:49,808 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-15 02:54:49,808 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-15 02:54:49,808 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-15 02:54:49,808 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-15 02:54:49,808 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-15 02:54:49,808 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-15 02:54:49,809 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:49,809 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:54:49,810 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-15 02:54:49,810 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-15 02:54:49,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-15 02:54:49,810 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:49,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:54:49,811 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:49,811 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-15 02:54:49,811 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:49,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-15 02:54:49,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:54:49,811 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-15 02:54:49,822 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-15 02:54:49,822 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-15 02:54:49,822 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-15 02:54:49,822 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-15 02:54:49,822 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-15 02:54:49,830 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-15 02:54:49,830 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681527274083.f8160290180ba126c904acc32b60eac7. 2023-04-15 02:54:49,830 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-15 02:54:49,830 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681527274083.f8160290180ba126c904acc32b60eac7. started... 2023-04-15 02:54:49,830 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681527274083.f8160290180ba126c904acc32b60eac7. as already flushing 2023-04-15 02:54:49,830 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681527274083.f8160290180ba126c904acc32b60eac7. 2023-04-15 02:54:49,830 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-15 02:54:49,830 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-15 02:54:49,830 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:49,830 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-15 02:54:49,830 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34681,1681527209133' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-15 02:54:49,832 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:49,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:49,832 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:49,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:54:49,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:54:49,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:54:49,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:54:49,833 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:49,833 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-15 02:54:49,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:54:49,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:49,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:54:49,835 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34681,1681527209133' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-15 02:54:49,835 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3d6d9c37[Count = 0] remaining members to acquire global barrier 2023-04-15 02:54:49,835 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-15 02:54:49,835 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:49,836 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:49,836 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:49,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:49,836 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-15 02:54:49,836 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-15 02:54:49,836 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34681,1681527209133' in zk 2023-04-15 02:54:49,837 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:49,837 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-15 02:54:49,837 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-15 02:54:49,837 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-15 02:54:49,837 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-15 02:54:49,838 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:49,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:49,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:54:49,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:54:49,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:54:49,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:54:49,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:54:49,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:49,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:54:49,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:54:49,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:49,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34681,1681527209133': 2023-04-15 02:54:49,842 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34681,1681527209133' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-15 02:54:49,842 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-15 02:54:49,842 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-15 02:54:49,842 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-15 02:54:49,842 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-15 02:54:49,842 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-15 02:54:49,844 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:49,844 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-15 02:54:49,844 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:49,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:49,844 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:49,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:49,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:54:49,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:54:49,845 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-15 02:54:49,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-15 02:54:49,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:54:49,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:49,846 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:49,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:54:49,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:54:49,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:54:49,847 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:49,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:49,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:54:49,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:54:49,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:49,857 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:49,858 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:54:49,858 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-15 02:54:49,858 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:54:49,858 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:49,858 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-15 02:54:49,858 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-15 02:54:49,858 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-15 02:54:49,858 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-15 02:54:49,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-15 02:54:49,858 DEBUG [Thread-1527] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-15 02:54:49,858 DEBUG [Thread-1527] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-15 02:54:49,858 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-15 02:54:49,858 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:49,858 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:49,858 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:49,858 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-15 02:54:49,858 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-15 02:54:49,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-15 02:54:49,958 DEBUG [Thread-1527] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-15 02:54:49,959 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-15 02:54:49,960 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-15 02:54:49,961 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-15 02:54:49,961 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-15 02:54:49,961 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-15 02:54:49,961 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-15 02:54:49,961 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-15 02:54:49,961 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-15 02:54:49,961 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:49,961 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:54:49,962 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-15 02:54:49,962 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:49,962 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-15 02:54:49,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-15 02:54:49,962 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:49,962 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-15 02:54:49,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:54:49,970 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:49,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-15 02:54:49,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:54:49,971 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-15 02:54:49,971 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-15 02:54:49,971 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-15 02:54:49,971 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-15 02:54:49,971 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-15 02:54:49,971 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-15 02:54:49,974 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-15 02:54:49,974 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-15 02:54:49,974 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681527274083.f8160290180ba126c904acc32b60eac7. 2023-04-15 02:54:49,974 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681527274083.f8160290180ba126c904acc32b60eac7. started... 2023-04-15 02:54:49,974 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681527274083.f8160290180ba126c904acc32b60eac7. as already flushing 2023-04-15 02:54:49,974 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681527274083.f8160290180ba126c904acc32b60eac7. 2023-04-15 02:54:49,974 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-15 02:54:49,974 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-15 02:54:49,974 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:49,974 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-15 02:54:49,974 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34681,1681527209133' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-15 02:54:49,975 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:49,975 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:49,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:49,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:54:49,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:54:49,975 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:49,976 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-15 02:54:49,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:54:49,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:54:49,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:54:49,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:49,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:54:49,976 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34681,1681527209133' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-15 02:54:49,977 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1ecfbe9d[Count = 0] remaining members to acquire global barrier 2023-04-15 02:54:49,977 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-15 02:54:49,977 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:49,977 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:49,977 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:49,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:49,977 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-15 02:54:49,977 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-15 02:54:49,977 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34681,1681527209133' in zk 2023-04-15 02:54:49,978 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:49,978 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-15 02:54:49,978 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:49,978 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-15 02:54:49,978 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-15 02:54:49,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:49,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:54:49,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:54:49,978 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-15 02:54:49,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:54:49,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:54:49,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:54:49,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:49,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:54:49,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:54:49,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:49,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34681,1681527209133': 2023-04-15 02:54:49,980 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34681,1681527209133' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-15 02:54:49,980 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-15 02:54:49,980 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-15 02:54:49,980 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-15 02:54:49,980 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-15 02:54:49,980 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-15 02:54:49,981 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:49,981 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:49,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:49,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:54:49,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:54:49,981 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:49,981 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-15 02:54:49,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:49,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:54:49,981 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-15 02:54:49,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-15 02:54:49,982 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:49,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:54:49,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:49,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:54:49,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:54:49,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:49,983 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:49,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:54:49,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:54:49,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:49,984 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:49,984 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-15 02:54:49,984 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:54:49,984 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-15 02:54:49,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-15 02:54:49,984 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:54:49,984 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-15 02:54:49,984 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-15 02:54:49,984 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-15 02:54:49,984 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-15 02:54:49,984 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:49,985 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:49,985 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:49,985 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:49,985 DEBUG [Thread-1527] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-15 02:54:49,985 DEBUG [Thread-1527] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-15 02:54:49,985 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-15 02:54:49,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-15 02:54:49,992 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f8160290180ba126c904acc32b60eac7, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:54:49,992 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34681] ipc.CallRunner(144): callId: 365 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:41918 deadline: 1681527349989, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f8160290180ba126c904acc32b60eac7, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:50,085 DEBUG [Thread-1527] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-15 02:54:50,085 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-15 02:54:50,087 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-15 02:54:50,087 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-15 02:54:50,087 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-15 02:54:50,087 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-15 02:54:50,088 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-15 02:54:50,088 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-15 02:54:50,088 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-15 02:54:50,088 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:50,088 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:54:50,090 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-15 02:54:50,090 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:50,090 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-15 02:54:50,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-15 02:54:50,090 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:50,090 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-15 02:54:50,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:54:50,090 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:50,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-15 02:54:50,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:54:50,091 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-15 02:54:50,091 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-15 02:54:50,091 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-15 02:54:50,091 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-15 02:54:50,091 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-15 02:54:50,091 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-15 02:54:50,091 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-15 02:54:50,091 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-15 02:54:50,091 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681527274083.f8160290180ba126c904acc32b60eac7. 2023-04-15 02:54:50,091 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681527274083.f8160290180ba126c904acc32b60eac7. started... 2023-04-15 02:54:50,091 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681527274083.f8160290180ba126c904acc32b60eac7. as already flushing 2023-04-15 02:54:50,091 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681527274083.f8160290180ba126c904acc32b60eac7. 2023-04-15 02:54:50,091 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-15 02:54:50,091 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-15 02:54:50,092 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:50,092 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-15 02:54:50,092 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34681,1681527209133' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-15 02:54:50,092 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:50,092 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:50,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:50,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:54:50,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:54:50,093 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:50,093 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-15 02:54:50,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:54:50,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:54:50,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:54:50,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:50,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:54:50,094 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34681,1681527209133' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-15 02:54:50,094 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-15 02:54:50,094 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6942c689[Count = 0] remaining members to acquire global barrier 2023-04-15 02:54:50,094 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:50,095 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:50,095 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:50,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:50,095 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-15 02:54:50,095 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-15 02:54:50,095 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:50,095 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-15 02:54:50,095 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34681,1681527209133' in zk 2023-04-15 02:54:50,096 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:50,096 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-15 02:54:50,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:50,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:54:50,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:54:50,096 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-15 02:54:50,096 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-15 02:54:50,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:54:50,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:54:50,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:54:50,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:50,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:54:50,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:54:50,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:50,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34681,1681527209133': 2023-04-15 02:54:50,097 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34681,1681527209133' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-15 02:54:50,097 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-15 02:54:50,098 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-15 02:54:50,098 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-15 02:54:50,098 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-15 02:54:50,098 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-15 02:54:50,098 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:50,098 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:50,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:50,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:54:50,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:54:50,098 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-15 02:54:50,098 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:50,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:50,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:54:50,099 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-15 02:54:50,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-15 02:54:50,099 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:50,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:50,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:54:50,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:54:50,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:54:50,100 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:50,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:50,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:54:50,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:54:50,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:50,101 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:50,101 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:54:50,101 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:54:50,101 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-15 02:54:50,101 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-15 02:54:50,101 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-15 02:54:50,101 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:50,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-15 02:54:50,101 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-15 02:54:50,101 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-15 02:54:50,101 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-15 02:54:50,101 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-15 02:54:50,102 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-15 02:54:50,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-15 02:54:50,101 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:50,102 DEBUG [Thread-1527] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-15 02:54:50,102 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:50,102 DEBUG [Thread-1527] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-15 02:54:50,102 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:50,202 DEBUG [Thread-1527] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-15 02:54:50,202 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-15 02:54:50,203 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-15 02:54:50,204 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-15 02:54:50,204 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-15 02:54:50,204 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-15 02:54:50,204 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-15 02:54:50,204 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-15 02:54:50,205 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-15 02:54:50,205 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:50,205 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:54:50,205 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-15 02:54:50,205 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:50,206 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-15 02:54:50,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-15 02:54:50,206 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:50,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:54:50,206 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-15 02:54:50,206 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:50,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-15 02:54:50,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:54:50,206 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-15 02:54:50,207 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-15 02:54:50,207 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-15 02:54:50,207 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-15 02:54:50,207 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-15 02:54:50,207 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-15 02:54:50,207 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-15 02:54:50,207 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-15 02:54:50,207 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681527274083.f8160290180ba126c904acc32b60eac7. 2023-04-15 02:54:50,207 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681527274083.f8160290180ba126c904acc32b60eac7. started... 2023-04-15 02:54:50,207 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681527274083.f8160290180ba126c904acc32b60eac7. as already flushing 2023-04-15 02:54:50,207 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681527274083.f8160290180ba126c904acc32b60eac7. 2023-04-15 02:54:50,207 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-15 02:54:50,207 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-15 02:54:50,207 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:50,207 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-15 02:54:50,207 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34681,1681527209133' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-15 02:54:50,208 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:50,208 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:50,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:50,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:54:50,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:54:50,208 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:50,208 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-15 02:54:50,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:54:50,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:54:50,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:54:50,210 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=49.20 KB at sequenceid=431 (bloomFilter=true), to=hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/f8160290180ba126c904acc32b60eac7/.tmp/B/cf9b4a5be1174051ab96d62fbd409586 2023-04-15 02:54:50,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:50,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:54:50,212 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34681,1681527209133' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-15 02:54:50,212 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@64360f3d[Count = 0] remaining members to acquire global barrier 2023-04-15 02:54:50,212 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-15 02:54:50,212 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:50,213 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:50,213 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:50,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:50,213 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-15 02:54:50,213 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-15 02:54:50,213 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34681,1681527209133' in zk 2023-04-15 02:54:50,213 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:50,213 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-15 02:54:50,214 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-15 02:54:50,214 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:50,214 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-15 02:54:50,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:50,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:54:50,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:54:50,214 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-15 02:54:50,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:54:50,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:54:50,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:54:50,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:50,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:54:50,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:54:50,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:50,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34681,1681527209133': 2023-04-15 02:54:50,218 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34681,1681527209133' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-15 02:54:50,218 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-15 02:54:50,218 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-15 02:54:50,218 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-15 02:54:50,218 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-15 02:54:50,218 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-15 02:54:50,224 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:50,224 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:50,224 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:50,224 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-15 02:54:50,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:50,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:50,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:54:50,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:54:50,224 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:50,225 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-15 02:54:50,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-15 02:54:50,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:54:50,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:50,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:54:50,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:54:50,225 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:50,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:54:50,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:50,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:54:50,228 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:50,228 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:54:50,228 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:54:50,228 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-15 02:54:50,228 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:50,228 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-15 02:54:50,228 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:50,228 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-15 02:54:50,228 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-15 02:54:50,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-15 02:54:50,228 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-15 02:54:50,228 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-15 02:54:50,228 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:50,229 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:50,229 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-15 02:54:50,229 DEBUG [Thread-1527] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-15 02:54:50,229 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-15 02:54:50,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-15 02:54:50,229 DEBUG [Thread-1527] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-15 02:54:50,329 DEBUG [Thread-1527] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-15 02:54:50,330 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-15 02:54:50,331 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-15 02:54:50,332 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-15 02:54:50,332 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-15 02:54:50,332 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-15 02:54:50,332 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-15 02:54:50,332 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-15 02:54:50,332 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-15 02:54:50,333 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:50,333 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:54:50,334 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:50,334 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-15 02:54:50,334 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-15 02:54:50,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-15 02:54:50,334 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:50,334 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-15 02:54:50,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:54:50,335 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:50,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-15 02:54:50,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:54:50,335 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-15 02:54:50,335 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-15 02:54:50,335 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-15 02:54:50,335 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-15 02:54:50,335 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-15 02:54:50,335 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-15 02:54:50,347 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-15 02:54:50,347 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-15 02:54:50,347 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681527274083.f8160290180ba126c904acc32b60eac7. 2023-04-15 02:54:50,347 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681527274083.f8160290180ba126c904acc32b60eac7. started... 2023-04-15 02:54:50,348 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681527274083.f8160290180ba126c904acc32b60eac7. as already flushing 2023-04-15 02:54:50,348 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681527274083.f8160290180ba126c904acc32b60eac7. 2023-04-15 02:54:50,348 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-15 02:54:50,348 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-15 02:54:50,348 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:50,348 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-15 02:54:50,348 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34681,1681527209133' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-15 02:54:50,349 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:50,349 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:50,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:50,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:54:50,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:54:50,349 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:50,349 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-15 02:54:50,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:54:50,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:54:50,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:54:50,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:50,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:54:50,350 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34681,1681527209133' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-15 02:54:50,350 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-15 02:54:50,350 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@356baba8[Count = 0] remaining members to acquire global barrier 2023-04-15 02:54:50,351 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:50,351 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:50,351 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:50,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:50,351 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-15 02:54:50,351 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-15 02:54:50,351 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34681,1681527209133' in zk 2023-04-15 02:54:50,351 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:50,351 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-15 02:54:50,352 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-15 02:54:50,352 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-15 02:54:50,352 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:50,352 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-15 02:54:50,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:50,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:54:50,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:54:50,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:54:50,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:54:50,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:54:50,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:50,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:54:50,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:54:50,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:50,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34681,1681527209133': 2023-04-15 02:54:50,354 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34681,1681527209133' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-15 02:54:50,354 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-15 02:54:50,354 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-15 02:54:50,354 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-15 02:54:50,354 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-15 02:54:50,354 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-15 02:54:50,355 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:50,355 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:50,355 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:50,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:50,355 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-15 02:54:50,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:50,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:54:50,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:54:50,356 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-15 02:54:50,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-15 02:54:50,356 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:50,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:54:50,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:50,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:54:50,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:54:50,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:54:50,357 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:50,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:50,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:54:50,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:54:50,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:50,358 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:50,358 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-15 02:54:50,358 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:54:50,358 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-15 02:54:50,358 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:54:50,358 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-15 02:54:50,358 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-15 02:54:50,358 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-15 02:54:50,358 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-15 02:54:50,358 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-15 02:54:50,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-15 02:54:50,358 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:50,358 DEBUG [Thread-1527] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-15 02:54:50,358 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:50,358 DEBUG [Thread-1527] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-15 02:54:50,359 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-15 02:54:50,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-15 02:54:50,359 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:50,359 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:50,459 DEBUG [Thread-1527] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-15 02:54:50,459 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-15 02:54:50,460 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-15 02:54:50,461 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-15 02:54:50,461 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-15 02:54:50,461 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-15 02:54:50,461 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-15 02:54:50,461 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-15 02:54:50,461 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-15 02:54:50,462 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:50,462 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:54:50,462 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-15 02:54:50,462 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:50,462 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-15 02:54:50,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-15 02:54:50,463 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:50,463 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-15 02:54:50,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:54:50,463 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:50,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-15 02:54:50,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:54:50,463 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-15 02:54:50,463 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-15 02:54:50,463 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-15 02:54:50,464 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-15 02:54:50,464 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-15 02:54:50,464 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-15 02:54:50,464 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-15 02:54:50,464 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-15 02:54:50,464 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681527274083.f8160290180ba126c904acc32b60eac7. 2023-04-15 02:54:50,464 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681527274083.f8160290180ba126c904acc32b60eac7. started... 2023-04-15 02:54:50,464 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681527274083.f8160290180ba126c904acc32b60eac7. as already flushing 2023-04-15 02:54:50,464 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681527274083.f8160290180ba126c904acc32b60eac7. 2023-04-15 02:54:50,464 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-15 02:54:50,464 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-15 02:54:50,464 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:50,464 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-15 02:54:50,464 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34681,1681527209133' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-15 02:54:50,465 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:50,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:50,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:54:50,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:54:50,465 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:50,465 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:50,466 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-15 02:54:50,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:54:50,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:54:50,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:54:50,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:50,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:54:50,467 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34681,1681527209133' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-15 02:54:50,467 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-15 02:54:50,467 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5633d0ce[Count = 0] remaining members to acquire global barrier 2023-04-15 02:54:50,467 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:50,468 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:50,468 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:50,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:50,468 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-15 02:54:50,468 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-15 02:54:50,468 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34681,1681527209133' in zk 2023-04-15 02:54:50,468 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:50,468 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-15 02:54:50,469 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:50,469 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-15 02:54:50,469 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-15 02:54:50,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:50,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:54:50,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:54:50,469 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-15 02:54:50,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:54:50,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:54:50,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:54:50,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:50,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:54:50,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:54:50,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:50,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34681,1681527209133': 2023-04-15 02:54:50,471 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34681,1681527209133' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-15 02:54:50,471 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-15 02:54:50,471 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-15 02:54:50,471 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-15 02:54:50,471 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-15 02:54:50,471 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-15 02:54:50,472 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:50,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:50,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:54:50,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:54:50,472 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:50,472 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-15 02:54:50,472 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:50,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:50,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:54:50,472 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:50,472 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-15 02:54:50,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-15 02:54:50,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:54:50,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:50,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:54:50,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:54:50,474 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:50,475 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-15 02:54:50,476 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-15 02:54:50,476 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:50,476 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-15 02:54:50,476 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-15 02:54:50,476 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:54:50,476 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-15 02:54:50,476 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-15 02:54:50,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-15 02:54:50,476 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:54:50,476 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:50,476 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:50,476 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:50,476 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:50,476 DEBUG [Thread-1527] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-15 02:54:50,476 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-15 02:54:50,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-15 02:54:50,476 DEBUG [Thread-1527] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-15 02:54:50,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:50,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:54:50,479 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-15 02:54:50,497 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f8160290180ba126c904acc32b60eac7, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:54:50,497 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34681] ipc.CallRunner(144): callId: 367 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:41918 deadline: 1681527350496, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f8160290180ba126c904acc32b60eac7, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:50,576 DEBUG [Thread-1527] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-15 02:54:50,577 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-15 02:54:50,578 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-15 02:54:50,579 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-15 02:54:50,579 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-15 02:54:50,579 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-15 02:54:50,579 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-15 02:54:50,579 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-15 02:54:50,579 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-15 02:54:50,579 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:50,579 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:54:50,580 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:50,580 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-15 02:54:50,580 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-15 02:54:50,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-15 02:54:50,580 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:50,580 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-15 02:54:50,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:54:50,581 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:50,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-15 02:54:50,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:54:50,581 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-15 02:54:50,581 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-15 02:54:50,581 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-15 02:54:50,581 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-15 02:54:50,581 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-15 02:54:50,581 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-15 02:54:50,582 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-15 02:54:50,582 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-15 02:54:50,582 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681527274083.f8160290180ba126c904acc32b60eac7. 2023-04-15 02:54:50,582 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681527274083.f8160290180ba126c904acc32b60eac7. started... 2023-04-15 02:54:50,582 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681527274083.f8160290180ba126c904acc32b60eac7. as already flushing 2023-04-15 02:54:50,582 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681527274083.f8160290180ba126c904acc32b60eac7. 2023-04-15 02:54:50,582 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-15 02:54:50,582 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-15 02:54:50,582 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:50,582 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-15 02:54:50,582 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34681,1681527209133' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-15 02:54:50,583 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:50,583 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:50,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:50,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:54:50,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:54:50,583 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:50,583 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-15 02:54:50,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:54:50,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:54:50,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:54:50,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:50,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:54:50,584 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34681,1681527209133' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-15 02:54:50,584 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-15 02:54:50,584 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@293a320f[Count = 0] remaining members to acquire global barrier 2023-04-15 02:54:50,584 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:50,585 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:50,585 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:50,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:50,585 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-15 02:54:50,585 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-15 02:54:50,585 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34681,1681527209133' in zk 2023-04-15 02:54:50,585 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:50,585 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-15 02:54:50,585 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:50,585 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-15 02:54:50,585 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-15 02:54:50,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:50,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:54:50,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:54:50,585 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-15 02:54:50,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:54:50,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:54:50,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:54:50,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:50,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:54:50,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:54:50,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:50,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34681,1681527209133': 2023-04-15 02:54:50,587 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34681,1681527209133' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-15 02:54:50,587 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-15 02:54:50,587 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-15 02:54:50,587 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-15 02:54:50,588 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-15 02:54:50,588 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-15 02:54:50,588 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:50,588 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:50,588 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:50,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:50,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:54:50,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:54:50,588 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-15 02:54:50,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:50,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:54:50,589 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-15 02:54:50,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-15 02:54:50,589 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:50,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:54:50,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:50,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:54:50,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:54:50,590 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:50,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:50,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:54:50,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:54:50,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:50,591 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-15 02:54:50,591 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-15 02:54:50,591 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-15 02:54:50,591 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:50,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-15 02:54:50,592 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-15 02:54:50,592 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-15 02:54:50,592 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-15 02:54:50,592 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-15 02:54:50,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-15 02:54:50,592 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-15 02:54:50,592 DEBUG [Thread-1527] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-15 02:54:50,592 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:54:50,592 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:54:50,592 DEBUG [Thread-1527] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-15 02:54:50,592 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:50,592 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:50,592 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:50,592 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:50,621 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=49.20 KB at sequenceid=431 (bloomFilter=true), to=hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/f8160290180ba126c904acc32b60eac7/.tmp/C/ce7ae2eeb5324a8fbb39fbe803efa5a7 2023-04-15 02:54:50,624 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/f8160290180ba126c904acc32b60eac7/.tmp/A/8ec77327a35847be8ff2bba511070406 as hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/f8160290180ba126c904acc32b60eac7/A/8ec77327a35847be8ff2bba511070406 2023-04-15 02:54:50,628 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/f8160290180ba126c904acc32b60eac7/A/8ec77327a35847be8ff2bba511070406, entries=150, sequenceid=431, filesize=11.9 K 2023-04-15 02:54:50,629 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/f8160290180ba126c904acc32b60eac7/.tmp/B/cf9b4a5be1174051ab96d62fbd409586 as hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/f8160290180ba126c904acc32b60eac7/B/cf9b4a5be1174051ab96d62fbd409586 2023-04-15 02:54:50,632 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/f8160290180ba126c904acc32b60eac7/B/cf9b4a5be1174051ab96d62fbd409586, entries=150, sequenceid=431, filesize=11.9 K 2023-04-15 02:54:50,632 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/f8160290180ba126c904acc32b60eac7/.tmp/C/ce7ae2eeb5324a8fbb39fbe803efa5a7 as hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/f8160290180ba126c904acc32b60eac7/C/ce7ae2eeb5324a8fbb39fbe803efa5a7 2023-04-15 02:54:50,636 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/f8160290180ba126c904acc32b60eac7/C/ce7ae2eeb5324a8fbb39fbe803efa5a7, entries=150, sequenceid=431, filesize=11.9 K 2023-04-15 02:54:50,636 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~147.60 KB/151140, heapSize ~422.11 KB/432240, currentSize=33.54 KB/34350 for f8160290180ba126c904acc32b60eac7 in 1282ms, sequenceid=431, compaction requested=true 2023-04-15 02:54:50,636 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for f8160290180ba126c904acc32b60eac7: 2023-04-15 02:54:50,637 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-15 02:54:50,637 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-15 02:54:50,637 DEBUG [RS:0;jenkins-hbase20:34681-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-15 02:54:50,637 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-15 02:54:50,637 DEBUG [RS:0;jenkins-hbase20:34681-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-15 02:54:50,638 DEBUG [RS:0;jenkins-hbase20:34681-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 34660 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-15 02:54:50,638 DEBUG [RS:0;jenkins-hbase20:34681-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 34660 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-15 02:54:50,638 DEBUG [RS:0;jenkins-hbase20:34681-longCompactions-0] regionserver.HStore(1912): f8160290180ba126c904acc32b60eac7/B is initiating minor compaction (all files) 2023-04-15 02:54:50,638 DEBUG [RS:0;jenkins-hbase20:34681-shortCompactions-0] regionserver.HStore(1912): f8160290180ba126c904acc32b60eac7/A is initiating minor compaction (all files) 2023-04-15 02:54:50,638 INFO [RS:0;jenkins-hbase20:34681-longCompactions-0] regionserver.HRegion(2259): Starting compaction of f8160290180ba126c904acc32b60eac7/B in TestAcidGuarantees,,1681527274083.f8160290180ba126c904acc32b60eac7. 2023-04-15 02:54:50,638 INFO [RS:0;jenkins-hbase20:34681-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of f8160290180ba126c904acc32b60eac7/A in TestAcidGuarantees,,1681527274083.f8160290180ba126c904acc32b60eac7. 2023-04-15 02:54:50,638 INFO [RS:0;jenkins-hbase20:34681-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/f8160290180ba126c904acc32b60eac7/B/4eca2f71b8374fbd86068b6b3f1136e8, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/f8160290180ba126c904acc32b60eac7/B/48c53c106b4642519b29b1e401289ef7, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/f8160290180ba126c904acc32b60eac7/B/cf9b4a5be1174051ab96d62fbd409586] into tmpdir=hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/f8160290180ba126c904acc32b60eac7/.tmp, totalSize=33.8 K 2023-04-15 02:54:50,638 INFO [RS:0;jenkins-hbase20:34681-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/f8160290180ba126c904acc32b60eac7/A/e73d4d3b96aa40cd914bc600fa1a7979, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/f8160290180ba126c904acc32b60eac7/A/0c0af3ff076a46db963ed21c3b7e3d42, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/f8160290180ba126c904acc32b60eac7/A/8ec77327a35847be8ff2bba511070406] into tmpdir=hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/f8160290180ba126c904acc32b60eac7/.tmp, totalSize=33.8 K 2023-04-15 02:54:50,638 DEBUG [RS:0;jenkins-hbase20:34681-longCompactions-0] compactions.Compactor(207): Compacting 4eca2f71b8374fbd86068b6b3f1136e8, keycount=150, bloomtype=ROW, size=12.4 K, encoding=NONE, compression=NONE, seqNum=313, earliestPutTs=1681527287039 2023-04-15 02:54:50,638 DEBUG [RS:0;jenkins-hbase20:34681-shortCompactions-0] compactions.Compactor(207): Compacting e73d4d3b96aa40cd914bc600fa1a7979, keycount=150, bloomtype=ROW, size=12.4 K, encoding=NONE, compression=NONE, seqNum=313, earliestPutTs=1681527287039 2023-04-15 02:54:50,639 DEBUG [RS:0;jenkins-hbase20:34681-longCompactions-0] compactions.Compactor(207): Compacting 48c53c106b4642519b29b1e401289ef7, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=322, earliestPutTs=1681527288207 2023-04-15 02:54:50,639 DEBUG [RS:0;jenkins-hbase20:34681-longCompactions-0] compactions.Compactor(207): Compacting cf9b4a5be1174051ab96d62fbd409586, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=431, earliestPutTs=1681527289235 2023-04-15 02:54:50,639 DEBUG [RS:0;jenkins-hbase20:34681-shortCompactions-0] compactions.Compactor(207): Compacting 0c0af3ff076a46db963ed21c3b7e3d42, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=322, earliestPutTs=1681527288207 2023-04-15 02:54:50,639 DEBUG [RS:0;jenkins-hbase20:34681-shortCompactions-0] compactions.Compactor(207): Compacting 8ec77327a35847be8ff2bba511070406, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=431, earliestPutTs=1681527289235 2023-04-15 02:54:50,659 INFO [RS:0;jenkins-hbase20:34681-longCompactions-0] throttle.PressureAwareThroughputController(145): f8160290180ba126c904acc32b60eac7#B#compaction#236 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-15 02:54:50,662 INFO [RS:0;jenkins-hbase20:34681-shortCompactions-0] throttle.PressureAwareThroughputController(145): f8160290180ba126c904acc32b60eac7#A#compaction#237 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-15 02:54:50,692 DEBUG [Thread-1527] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-15 02:54:50,693 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-15 02:54:50,694 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-15 02:54:50,698 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-15 02:54:50,698 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-15 02:54:50,698 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-15 02:54:50,698 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-15 02:54:50,698 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-15 02:54:50,698 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-15 02:54:50,699 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:50,699 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:54:50,701 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-15 02:54:50,701 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-15 02:54:50,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-15 02:54:50,701 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:50,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:54:50,701 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:50,701 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-15 02:54:50,702 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:50,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-15 02:54:50,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:54:50,702 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-15 02:54:50,702 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-15 02:54:50,703 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-15 02:54:50,703 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-15 02:54:50,703 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-15 02:54:50,703 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-15 02:54:50,703 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-15 02:54:50,703 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-15 02:54:50,703 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681527274083.f8160290180ba126c904acc32b60eac7. 2023-04-15 02:54:50,703 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681527274083.f8160290180ba126c904acc32b60eac7. started... 2023-04-15 02:54:50,703 INFO [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing f8160290180ba126c904acc32b60eac7 3/3 column families, dataSize=33.54 KB heapSize=100.36 KB 2023-04-15 02:54:50,703 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK f8160290180ba126c904acc32b60eac7, store=A 2023-04-15 02:54:50,703 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-15 02:54:50,703 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK f8160290180ba126c904acc32b60eac7, store=B 2023-04-15 02:54:50,703 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-15 02:54:50,704 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK f8160290180ba126c904acc32b60eac7, store=C 2023-04-15 02:54:50,704 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-15 02:54:50,704 DEBUG [RS:0;jenkins-hbase20:34681-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/f8160290180ba126c904acc32b60eac7/.tmp/B/a87b02c9f45148c6b5788737b0ebb158 as hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/f8160290180ba126c904acc32b60eac7/B/a87b02c9f45148c6b5788737b0ebb158 2023-04-15 02:54:50,708 INFO [RS:0;jenkins-hbase20:34681-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in f8160290180ba126c904acc32b60eac7/B of f8160290180ba126c904acc32b60eac7 into a87b02c9f45148c6b5788737b0ebb158(size=12.5 K), total size for store is 12.5 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-15 02:54:50,708 DEBUG [RS:0;jenkins-hbase20:34681-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for f8160290180ba126c904acc32b60eac7: 2023-04-15 02:54:50,708 INFO [RS:0;jenkins-hbase20:34681-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681527274083.f8160290180ba126c904acc32b60eac7., storeName=f8160290180ba126c904acc32b60eac7/B, priority=13, startTime=1681527290637; duration=0sec 2023-04-15 02:54:50,708 DEBUG [RS:0;jenkins-hbase20:34681-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-15 02:54:50,708 DEBUG [RS:0;jenkins-hbase20:34681-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-15 02:54:50,709 DEBUG [RS:0;jenkins-hbase20:34681-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 34660 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-15 02:54:50,709 DEBUG [RS:0;jenkins-hbase20:34681-longCompactions-0] regionserver.HStore(1912): f8160290180ba126c904acc32b60eac7/C is initiating minor compaction (all files) 2023-04-15 02:54:50,709 INFO [RS:0;jenkins-hbase20:34681-longCompactions-0] regionserver.HRegion(2259): Starting compaction of f8160290180ba126c904acc32b60eac7/C in TestAcidGuarantees,,1681527274083.f8160290180ba126c904acc32b60eac7. 2023-04-15 02:54:50,709 INFO [RS:0;jenkins-hbase20:34681-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/f8160290180ba126c904acc32b60eac7/C/378b0b8eb7bd42889792cec86a396fda, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/f8160290180ba126c904acc32b60eac7/C/a6113b6a91f3426c92e0b038b548780a, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/f8160290180ba126c904acc32b60eac7/C/ce7ae2eeb5324a8fbb39fbe803efa5a7] into tmpdir=hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/f8160290180ba126c904acc32b60eac7/.tmp, totalSize=33.8 K 2023-04-15 02:54:50,710 DEBUG [RS:0;jenkins-hbase20:34681-longCompactions-0] compactions.Compactor(207): Compacting 378b0b8eb7bd42889792cec86a396fda, keycount=150, bloomtype=ROW, size=12.4 K, encoding=NONE, compression=NONE, seqNum=313, earliestPutTs=1681527287039 2023-04-15 02:54:50,710 DEBUG [RS:0;jenkins-hbase20:34681-longCompactions-0] compactions.Compactor(207): Compacting a6113b6a91f3426c92e0b038b548780a, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=322, earliestPutTs=1681527288207 2023-04-15 02:54:50,711 DEBUG [RS:0;jenkins-hbase20:34681-longCompactions-0] compactions.Compactor(207): Compacting ce7ae2eeb5324a8fbb39fbe803efa5a7, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=431, earliestPutTs=1681527289235 2023-04-15 02:54:50,714 INFO [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=440 (bloomFilter=true), to=hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/f8160290180ba126c904acc32b60eac7/.tmp/A/88d467e58ea44e1fbf4749826d79cc70 2023-04-15 02:54:50,723 INFO [RS:0;jenkins-hbase20:34681-longCompactions-0] throttle.PressureAwareThroughputController(145): f8160290180ba126c904acc32b60eac7#C#compaction#239 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-15 02:54:51,104 DEBUG [RS:0;jenkins-hbase20:34681-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/f8160290180ba126c904acc32b60eac7/.tmp/A/ed9f1352085e4cd4a6046951a999c3fb as hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/f8160290180ba126c904acc32b60eac7/A/ed9f1352085e4cd4a6046951a999c3fb 2023-04-15 02:54:51,109 INFO [RS:0;jenkins-hbase20:34681-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in f8160290180ba126c904acc32b60eac7/A of f8160290180ba126c904acc32b60eac7 into ed9f1352085e4cd4a6046951a999c3fb(size=12.5 K), total size for store is 12.5 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-15 02:54:51,109 DEBUG [RS:0;jenkins-hbase20:34681-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for f8160290180ba126c904acc32b60eac7: 2023-04-15 02:54:51,109 INFO [RS:0;jenkins-hbase20:34681-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681527274083.f8160290180ba126c904acc32b60eac7., storeName=f8160290180ba126c904acc32b60eac7/A, priority=13, startTime=1681527290637; duration=0sec 2023-04-15 02:54:51,109 DEBUG [RS:0;jenkins-hbase20:34681-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-15 02:54:51,128 INFO [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=440 (bloomFilter=true), to=hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/f8160290180ba126c904acc32b60eac7/.tmp/B/ea1cf09ab444475b96e383a56dc20362 2023-04-15 02:54:51,143 DEBUG [RS:0;jenkins-hbase20:34681-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/f8160290180ba126c904acc32b60eac7/.tmp/C/40a96079c22f4d509d57c890e94b1486 as hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/f8160290180ba126c904acc32b60eac7/C/40a96079c22f4d509d57c890e94b1486 2023-04-15 02:54:51,144 INFO [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=440 (bloomFilter=true), to=hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/f8160290180ba126c904acc32b60eac7/.tmp/C/0449ab308b884ddf8b41d677a28aff3f 2023-04-15 02:54:51,149 INFO [RS:0;jenkins-hbase20:34681-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in f8160290180ba126c904acc32b60eac7/C of f8160290180ba126c904acc32b60eac7 into 40a96079c22f4d509d57c890e94b1486(size=12.5 K), total size for store is 12.5 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-15 02:54:51,149 DEBUG [RS:0;jenkins-hbase20:34681-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for f8160290180ba126c904acc32b60eac7: 2023-04-15 02:54:51,149 INFO [RS:0;jenkins-hbase20:34681-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681527274083.f8160290180ba126c904acc32b60eac7., storeName=f8160290180ba126c904acc32b60eac7/C, priority=13, startTime=1681527290637; duration=0sec 2023-04-15 02:54:51,149 DEBUG [RS:0;jenkins-hbase20:34681-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-15 02:54:51,151 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/f8160290180ba126c904acc32b60eac7/.tmp/A/88d467e58ea44e1fbf4749826d79cc70 as hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/f8160290180ba126c904acc32b60eac7/A/88d467e58ea44e1fbf4749826d79cc70 2023-04-15 02:54:51,155 INFO [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/f8160290180ba126c904acc32b60eac7/A/88d467e58ea44e1fbf4749826d79cc70, entries=150, sequenceid=440, filesize=11.9 K 2023-04-15 02:54:51,156 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/f8160290180ba126c904acc32b60eac7/.tmp/B/ea1cf09ab444475b96e383a56dc20362 as hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/f8160290180ba126c904acc32b60eac7/B/ea1cf09ab444475b96e383a56dc20362 2023-04-15 02:54:51,161 INFO [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/f8160290180ba126c904acc32b60eac7/B/ea1cf09ab444475b96e383a56dc20362, entries=150, sequenceid=440, filesize=11.9 K 2023-04-15 02:54:51,162 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/f8160290180ba126c904acc32b60eac7/.tmp/C/0449ab308b884ddf8b41d677a28aff3f as hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/f8160290180ba126c904acc32b60eac7/C/0449ab308b884ddf8b41d677a28aff3f 2023-04-15 02:54:51,168 INFO [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/f8160290180ba126c904acc32b60eac7/C/0449ab308b884ddf8b41d677a28aff3f, entries=150, sequenceid=440, filesize=11.9 K 2023-04-15 02:54:51,169 INFO [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~33.54 KB/34350, heapSize ~100.31 KB/102720, currentSize=0 B/0 for f8160290180ba126c904acc32b60eac7 in 466ms, sequenceid=440, compaction requested=false 2023-04-15 02:54:51,169 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for f8160290180ba126c904acc32b60eac7: 2023-04-15 02:54:51,169 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681527274083.f8160290180ba126c904acc32b60eac7. 2023-04-15 02:54:51,169 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-15 02:54:51,169 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-15 02:54:51,169 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:51,169 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-15 02:54:51,169 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34681,1681527209133' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-15 02:54:51,170 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:51,170 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:51,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:51,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:54:51,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:54:51,170 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:51,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:54:51,171 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-15 02:54:51,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:54:51,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:54:51,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:51,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:54:51,171 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34681,1681527209133' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-15 02:54:51,172 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-15 02:54:51,172 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7da18ff2[Count = 0] remaining members to acquire global barrier 2023-04-15 02:54:51,172 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:51,172 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:51,172 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:51,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:51,172 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-15 02:54:51,172 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:51,172 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-15 02:54:51,172 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-15 02:54:51,172 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34681,1681527209133' in zk 2023-04-15 02:54:51,173 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-15 02:54:51,173 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-15 02:54:51,173 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-15 02:54:51,173 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:51,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:51,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:54:51,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:54:51,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:54:51,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:54:51,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:54:51,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:51,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:54:51,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:54:51,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:51,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34681,1681527209133': 2023-04-15 02:54:51,176 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34681,1681527209133' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-15 02:54:51,176 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-15 02:54:51,176 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-15 02:54:51,176 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-15 02:54:51,176 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-15 02:54:51,176 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-15 02:54:51,177 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:51,177 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:51,177 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:51,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:51,177 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-15 02:54:51,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:51,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:54:51,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:54:51,178 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-15 02:54:51,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-15 02:54:51,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:54:51,178 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:51,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:51,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:54:51,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:54:51,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:54:51,179 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:51,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:51,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:54:51,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:54:51,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:51,181 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:51,181 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:54:51,181 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:54:51,181 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-15 02:54:51,181 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:51,181 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-15 02:54:51,181 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-15 02:54:51,181 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:51,182 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:51,182 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:51,182 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-15 02:54:51,182 DEBUG [Thread-1527] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-15 02:54:51,182 DEBUG [Thread-1527] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-15 02:54:51,182 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-15 02:54:51,182 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-15 02:54:51,182 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-15 02:54:51,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-15 02:54:51,183 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-15 02:54:51,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-15 02:54:51,288 DEBUG [Thread-1527] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-15 02:54:51,288 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-15 02:54:51,290 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-15 02:54:51,292 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-15 02:54:51,292 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-15 02:54:51,292 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-15 02:54:51,292 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-15 02:54:51,292 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-15 02:54:51,292 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-15 02:54:51,292 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:51,292 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:54:51,293 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-15 02:54:51,293 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:51,293 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-15 02:54:51,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-15 02:54:51,293 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:51,293 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-15 02:54:51,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:54:51,294 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:51,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-15 02:54:51,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:54:51,295 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-15 02:54:51,295 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-15 02:54:51,295 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-15 02:54:51,295 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-15 02:54:51,295 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-15 02:54:51,296 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-15 02:54:51,296 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681527274083.f8160290180ba126c904acc32b60eac7. 2023-04-15 02:54:51,296 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-15 02:54:51,296 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681527274083.f8160290180ba126c904acc32b60eac7. started... 2023-04-15 02:54:51,297 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for f8160290180ba126c904acc32b60eac7: 2023-04-15 02:54:51,297 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681527274083.f8160290180ba126c904acc32b60eac7. 2023-04-15 02:54:51,297 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-15 02:54:51,297 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-15 02:54:51,297 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:51,297 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-15 02:54:51,297 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34681,1681527209133' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-15 02:54:51,298 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:51,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:51,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:54:51,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:54:51,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:54:51,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:54:51,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:54:51,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:51,299 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:51,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:54:51,300 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:51,300 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-15 02:54:51,301 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34681,1681527209133' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-15 02:54:51,301 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@157f5d05[Count = 0] remaining members to acquire global barrier 2023-04-15 02:54:51,301 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-15 02:54:51,301 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:51,302 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:51,302 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:51,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:51,302 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-15 02:54:51,302 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:51,302 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-15 02:54:51,302 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-15 02:54:51,302 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34681,1681527209133' in zk 2023-04-15 02:54:51,304 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:51,304 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-15 02:54:51,304 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-15 02:54:51,304 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-15 02:54:51,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:51,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:54:51,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:54:51,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:54:51,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:54:51,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:54:51,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:51,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:54:51,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:54:51,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:51,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34681,1681527209133': 2023-04-15 02:54:51,306 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34681,1681527209133' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-15 02:54:51,306 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-15 02:54:51,306 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-15 02:54:51,306 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-15 02:54:51,306 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-15 02:54:51,307 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-15 02:54:51,308 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:51,308 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-15 02:54:51,308 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:51,308 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:51,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:51,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:51,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:54:51,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:54:51,308 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-15 02:54:51,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-15 02:54:51,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:51,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:54:51,309 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:51,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:54:51,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:54:51,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:54:51,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:51,311 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:51,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:54:51,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:54:51,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:51,314 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-15 02:54:51,314 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-15 02:54:51,314 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-15 02:54:51,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-15 02:54:51,314 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-15 02:54:51,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-15 02:54:51,314 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:51,314 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-15 02:54:51,314 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-15 02:54:51,314 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:54:51,315 DEBUG [Thread-1527] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-15 02:54:51,315 DEBUG [Thread-1527] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-15 02:54:51,314 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-15 02:54:51,315 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:54:51,315 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:51,315 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:51,315 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:51,315 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:51,415 DEBUG [Thread-1527] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-15 02:54:51,415 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-15 02:54:51,417 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-15 02:54:51,418 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-15 02:54:51,418 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-15 02:54:51,418 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-15 02:54:51,418 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-15 02:54:51,418 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-15 02:54:51,418 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-15 02:54:51,419 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:51,419 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:54:51,419 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:51,419 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-15 02:54:51,419 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-15 02:54:51,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-15 02:54:51,420 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:51,420 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-15 02:54:51,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:54:51,420 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:51,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-15 02:54:51,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:54:51,421 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-15 02:54:51,422 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-15 02:54:51,422 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-15 02:54:51,422 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-15 02:54:51,422 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-15 02:54:51,422 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-15 02:54:51,422 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-15 02:54:51,422 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-15 02:54:51,422 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681527274083.f8160290180ba126c904acc32b60eac7. 2023-04-15 02:54:51,422 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681527274083.f8160290180ba126c904acc32b60eac7. started... 2023-04-15 02:54:51,422 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for f8160290180ba126c904acc32b60eac7: 2023-04-15 02:54:51,422 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681527274083.f8160290180ba126c904acc32b60eac7. 2023-04-15 02:54:51,422 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-15 02:54:51,423 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-15 02:54:51,423 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:51,423 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-15 02:54:51,423 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34681,1681527209133' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-15 02:54:51,423 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:51,424 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:51,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:51,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:54:51,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:54:51,424 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:51,424 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-15 02:54:51,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:54:51,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:54:51,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:54:51,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:51,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:54:51,425 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34681,1681527209133' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-15 02:54:51,425 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5d825ac4[Count = 0] remaining members to acquire global barrier 2023-04-15 02:54:51,425 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-15 02:54:51,425 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:51,425 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:51,426 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:51,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:51,426 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-15 02:54:51,426 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-15 02:54:51,426 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34681,1681527209133' in zk 2023-04-15 02:54:51,426 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:51,426 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-15 02:54:51,426 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:51,426 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-15 02:54:51,426 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-15 02:54:51,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:51,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:54:51,426 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-15 02:54:51,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:54:51,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:54:51,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:54:51,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:54:51,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:51,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:54:51,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:54:51,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:51,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34681,1681527209133': 2023-04-15 02:54:51,429 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34681,1681527209133' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-15 02:54:51,429 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-15 02:54:51,429 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-15 02:54:51,429 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-15 02:54:51,429 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-15 02:54:51,429 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-15 02:54:51,430 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:51,430 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:51,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:51,430 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:51,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:54:51,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:54:51,430 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-15 02:54:51,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:51,430 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:51,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:54:51,430 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-15 02:54:51,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-15 02:54:51,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:51,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:54:51,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:54:51,431 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:51,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:54:51,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:51,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:54:51,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:54:51,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:51,432 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:51,433 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:54:51,433 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-15 02:54:51,433 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-15 02:54:51,433 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-15 02:54:51,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-15 02:54:51,433 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-15 02:54:51,433 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-15 02:54:51,433 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-15 02:54:51,433 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:54:51,433 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-15 02:54:51,433 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:51,433 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:51,433 DEBUG [Thread-1527] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-15 02:54:51,433 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-15 02:54:51,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-15 02:54:51,433 DEBUG [Thread-1527] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-15 02:54:51,433 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:51,433 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:51,516 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34681] regionserver.HRegion(9158): Flush requested on f8160290180ba126c904acc32b60eac7 2023-04-15 02:54:51,517 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing f8160290180ba126c904acc32b60eac7 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-15 02:54:51,517 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK f8160290180ba126c904acc32b60eac7, store=A 2023-04-15 02:54:51,517 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-15 02:54:51,517 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK f8160290180ba126c904acc32b60eac7, store=B 2023-04-15 02:54:51,517 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-15 02:54:51,517 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK f8160290180ba126c904acc32b60eac7, store=C 2023-04-15 02:54:51,517 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-15 02:54:51,533 DEBUG [Thread-1527] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-15 02:54:51,534 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-15 02:54:51,535 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-15 02:54:51,536 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-15 02:54:51,536 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-15 02:54:51,536 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-15 02:54:51,536 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-15 02:54:51,536 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-15 02:54:51,536 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-15 02:54:51,536 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:51,536 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:54:51,537 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-15 02:54:51,537 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:51,537 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-15 02:54:51,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-15 02:54:51,537 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:51,537 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-15 02:54:51,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:54:51,537 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:51,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-15 02:54:51,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:54:51,538 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-15 02:54:51,538 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-15 02:54:51,538 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-15 02:54:51,538 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-15 02:54:51,538 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-15 02:54:51,538 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-15 02:54:51,538 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-15 02:54:51,538 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-15 02:54:51,538 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681527274083.f8160290180ba126c904acc32b60eac7. 2023-04-15 02:54:51,538 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681527274083.f8160290180ba126c904acc32b60eac7. started... 2023-04-15 02:54:51,538 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681527274083.f8160290180ba126c904acc32b60eac7. as already flushing 2023-04-15 02:54:51,538 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681527274083.f8160290180ba126c904acc32b60eac7. 2023-04-15 02:54:51,538 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-15 02:54:51,538 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-15 02:54:51,538 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:51,539 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-15 02:54:51,539 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34681,1681527209133' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-15 02:54:51,539 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:51,539 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:51,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:51,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:54:51,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:54:51,539 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:51,539 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-15 02:54:51,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:54:51,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:54:51,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:54:51,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:51,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:54:51,541 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34681,1681527209133' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-15 02:54:51,541 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2550a131[Count = 0] remaining members to acquire global barrier 2023-04-15 02:54:51,541 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-15 02:54:51,541 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:51,541 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:51,541 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:51,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:51,541 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-15 02:54:51,541 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-15 02:54:51,541 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34681,1681527209133' in zk 2023-04-15 02:54:51,541 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:51,542 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-15 02:54:51,542 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:51,542 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-15 02:54:51,542 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-15 02:54:51,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:51,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:54:51,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:54:51,542 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-15 02:54:51,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:54:51,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:54:51,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:54:51,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:51,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:54:51,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:54:51,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:51,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34681,1681527209133': 2023-04-15 02:54:51,544 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34681,1681527209133' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-15 02:54:51,544 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-15 02:54:51,544 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-15 02:54:51,544 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-15 02:54:51,544 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-15 02:54:51,544 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-15 02:54:51,545 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:51,545 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:51,545 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:51,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:51,545 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-15 02:54:51,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:51,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:54:51,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:54:51,545 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-15 02:54:51,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-15 02:54:51,546 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:51,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:54:51,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:51,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:54:51,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:54:51,546 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:51,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:54:51,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:51,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:54:51,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:54:51,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:51,548 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:51,548 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:54:51,548 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:54:51,548 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:51,548 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-15 02:54:51,548 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-15 02:54:51,548 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-15 02:54:51,548 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-15 02:54:51,548 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-15 02:54:51,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-15 02:54:51,548 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-15 02:54:51,548 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:51,548 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:51,548 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:51,549 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-15 02:54:51,549 DEBUG [Thread-1527] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-15 02:54:51,549 DEBUG [Thread-1527] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-15 02:54:51,549 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-15 02:54:51,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-15 02:54:51,556 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681527291537, max timestamp=1681527291555Num uniques 859; 2023-04-15 02:54:51,556 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-5] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-15 02:54:51,557 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681527291537, max timestamp=1681527291555Num uniques 859; 2023-04-15 02:54:51,557 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681527291537, max timestamp=1681527291555Num uniques 859; 2023-04-15 02:54:51,557 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-6] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-15 02:54:51,557 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-4] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-15 02:54:51,606 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f8160290180ba126c904acc32b60eac7, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:54:51,606 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34681] ipc.CallRunner(144): callId: 410 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:41918 deadline: 1681527351604, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f8160290180ba126c904acc32b60eac7, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:51,649 DEBUG [Thread-1527] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-15 02:54:51,649 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-15 02:54:51,650 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-15 02:54:51,651 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-15 02:54:51,651 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-15 02:54:51,651 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-15 02:54:51,651 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-15 02:54:51,651 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-15 02:54:51,651 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-15 02:54:51,652 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:51,652 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:54:51,652 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-15 02:54:51,652 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:51,652 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-15 02:54:51,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-15 02:54:51,653 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:51,653 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-15 02:54:51,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:54:51,653 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:51,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-15 02:54:51,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:54:51,653 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-15 02:54:51,653 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-15 02:54:51,653 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-15 02:54:51,653 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-15 02:54:51,654 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-15 02:54:51,654 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-15 02:54:51,654 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-15 02:54:51,654 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-15 02:54:51,654 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681527274083.f8160290180ba126c904acc32b60eac7. 2023-04-15 02:54:51,654 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681527274083.f8160290180ba126c904acc32b60eac7. started... 2023-04-15 02:54:51,654 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681527274083.f8160290180ba126c904acc32b60eac7. as already flushing 2023-04-15 02:54:51,654 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681527274083.f8160290180ba126c904acc32b60eac7. 2023-04-15 02:54:51,654 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-15 02:54:51,654 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-15 02:54:51,654 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:51,654 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-15 02:54:51,654 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34681,1681527209133' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-15 02:54:51,655 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:51,655 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:51,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:51,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:54:51,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:54:51,656 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:51,656 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-15 02:54:51,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:54:51,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:54:51,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:54:51,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:51,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:54:51,661 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34681,1681527209133' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-15 02:54:51,661 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-15 02:54:51,661 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@54ff7476[Count = 0] remaining members to acquire global barrier 2023-04-15 02:54:51,661 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:51,670 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:51,670 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:51,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:51,670 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:51,670 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-15 02:54:51,670 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-15 02:54:51,670 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-15 02:54:51,670 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34681,1681527209133' in zk 2023-04-15 02:54:51,674 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:51,674 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-15 02:54:51,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:51,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:54:51,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:54:51,674 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-15 02:54:51,674 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-15 02:54:51,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:54:51,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:54:51,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:54:51,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:51,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:54:51,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:54:51,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:51,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34681,1681527209133': 2023-04-15 02:54:51,676 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34681,1681527209133' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-15 02:54:51,676 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-15 02:54:51,676 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-15 02:54:51,676 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-15 02:54:51,676 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-15 02:54:51,676 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-15 02:54:51,678 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:51,678 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:51,678 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-15 02:54:51,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:51,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:54:51,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:54:51,678 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:51,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:51,679 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:51,679 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-15 02:54:51,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-15 02:54:51,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:54:51,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:51,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:54:51,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:54:51,680 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:51,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:54:51,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:51,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:54:51,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:54:51,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:51,682 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:51,682 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-15 02:54:51,682 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:54:51,682 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-15 02:54:51,682 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-15 02:54:51,682 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:54:51,682 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-15 02:54:51,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-15 02:54:51,682 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-15 02:54:51,683 DEBUG [Thread-1527] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-15 02:54:51,682 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-15 02:54:51,683 DEBUG [Thread-1527] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-15 02:54:51,683 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-15 02:54:51,682 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:51,683 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:51,683 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-15 02:54:51,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-15 02:54:51,683 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:51,683 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:51,708 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f8160290180ba126c904acc32b60eac7, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:54:51,708 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34681] ipc.CallRunner(144): callId: 412 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:41918 deadline: 1681527351707, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f8160290180ba126c904acc32b60eac7, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:51,783 DEBUG [Thread-1527] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-15 02:54:51,783 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-15 02:54:51,785 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-15 02:54:51,786 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-15 02:54:51,786 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-15 02:54:51,786 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-15 02:54:51,786 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-15 02:54:51,786 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-15 02:54:51,786 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-15 02:54:51,786 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:51,786 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:54:51,787 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-15 02:54:51,787 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:51,787 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-15 02:54:51,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-15 02:54:51,787 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:51,787 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-15 02:54:51,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:54:51,788 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:51,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-15 02:54:51,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:54:51,788 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-15 02:54:51,788 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-15 02:54:51,788 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-15 02:54:51,789 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-15 02:54:51,789 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-15 02:54:51,789 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-15 02:54:51,790 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-15 02:54:51,790 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-15 02:54:51,790 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681527274083.f8160290180ba126c904acc32b60eac7. 2023-04-15 02:54:51,790 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681527274083.f8160290180ba126c904acc32b60eac7. started... 2023-04-15 02:54:51,790 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681527274083.f8160290180ba126c904acc32b60eac7. as already flushing 2023-04-15 02:54:51,790 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681527274083.f8160290180ba126c904acc32b60eac7. 2023-04-15 02:54:51,790 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-15 02:54:51,790 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-15 02:54:51,790 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:51,790 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-15 02:54:51,790 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34681,1681527209133' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-15 02:54:51,791 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:51,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:51,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:54:51,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:54:51,791 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:51,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:54:51,792 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:51,792 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:54:51,792 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-15 02:54:51,792 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:54:51,792 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:51,792 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:54:51,792 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34681,1681527209133' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-15 02:54:51,792 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3b078c99[Count = 0] remaining members to acquire global barrier 2023-04-15 02:54:51,792 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-15 02:54:51,792 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:51,793 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:51,793 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:51,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:51,793 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-15 02:54:51,793 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-15 02:54:51,793 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34681,1681527209133' in zk 2023-04-15 02:54:51,793 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:51,793 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-15 02:54:51,794 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:51,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:51,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:54:51,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:54:51,794 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-15 02:54:51,794 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-15 02:54:51,794 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-15 02:54:51,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:54:51,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:54:51,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:54:51,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:51,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:54:51,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:54:51,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:51,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34681,1681527209133': 2023-04-15 02:54:51,796 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34681,1681527209133' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-15 02:54:51,796 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-15 02:54:51,796 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-15 02:54:51,796 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-15 02:54:51,796 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-15 02:54:51,796 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-15 02:54:51,797 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:51,797 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:51,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:51,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:54:51,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:54:51,797 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:51,797 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-15 02:54:51,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:51,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:54:51,797 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:51,797 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-15 02:54:51,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-15 02:54:51,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:54:51,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:51,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:54:51,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:54:51,798 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:51,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:51,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:54:51,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:54:51,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:51,799 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:51,799 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:54:51,799 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-15 02:54:51,799 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:54:51,800 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-15 02:54:51,800 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-15 02:54:51,800 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-15 02:54:51,800 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:51,800 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:51,800 DEBUG [Thread-1527] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-15 02:54:51,800 DEBUG [Thread-1527] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-15 02:54:51,800 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-15 02:54:51,800 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-15 02:54:51,800 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-15 02:54:51,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-15 02:54:51,800 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:51,800 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:51,800 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-15 02:54:51,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-15 02:54:51,900 DEBUG [Thread-1527] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-15 02:54:51,900 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-15 02:54:51,902 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-15 02:54:51,904 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-15 02:54:51,904 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-15 02:54:51,904 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-15 02:54:51,904 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-15 02:54:51,904 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-15 02:54:51,904 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-15 02:54:51,905 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:51,905 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:54:51,905 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:51,905 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-15 02:54:51,905 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-15 02:54:51,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-15 02:54:51,906 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:51,906 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-15 02:54:51,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:54:51,906 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:51,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-15 02:54:51,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:54:51,907 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-15 02:54:51,907 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-15 02:54:51,907 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-15 02:54:51,907 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-15 02:54:51,907 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-15 02:54:51,907 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-15 02:54:51,908 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-15 02:54:51,908 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-15 02:54:51,908 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681527274083.f8160290180ba126c904acc32b60eac7. 2023-04-15 02:54:51,908 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681527274083.f8160290180ba126c904acc32b60eac7. started... 2023-04-15 02:54:51,908 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681527274083.f8160290180ba126c904acc32b60eac7. as already flushing 2023-04-15 02:54:51,908 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681527274083.f8160290180ba126c904acc32b60eac7. 2023-04-15 02:54:51,908 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-15 02:54:51,908 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-15 02:54:51,908 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:51,908 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-15 02:54:51,908 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34681,1681527209133' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-15 02:54:51,909 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:51,909 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:51,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:51,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:54:51,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:54:51,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:54:51,910 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:51,910 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-15 02:54:51,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:54:51,911 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f8160290180ba126c904acc32b60eac7, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:54:51,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:54:51,911 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34681] ipc.CallRunner(144): callId: 414 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:41918 deadline: 1681527351910, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f8160290180ba126c904acc32b60eac7, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:51,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:51,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:54:51,911 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34681,1681527209133' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-15 02:54:51,911 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@749cb547[Count = 0] remaining members to acquire global barrier 2023-04-15 02:54:51,911 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-15 02:54:51,911 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:51,912 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:51,912 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:51,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:51,912 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-15 02:54:51,912 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-15 02:54:51,912 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:51,912 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-15 02:54:51,912 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34681,1681527209133' in zk 2023-04-15 02:54:51,913 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:51,913 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-15 02:54:51,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:51,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:54:51,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:54:51,913 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-15 02:54:51,913 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-15 02:54:51,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:54:51,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:54:51,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:54:51,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:51,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:54:51,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:54:51,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:51,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34681,1681527209133': 2023-04-15 02:54:51,915 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34681,1681527209133' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-15 02:54:51,915 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-15 02:54:51,915 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-15 02:54:51,915 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-15 02:54:51,915 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-15 02:54:51,915 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-15 02:54:51,915 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:51,916 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:51,916 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-15 02:54:51,916 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:51,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:51,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:51,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:54:51,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:54:51,916 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-15 02:54:51,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-15 02:54:51,916 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:51,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:51,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:54:51,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:54:51,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:54:51,917 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:51,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:54:51,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:51,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:54:51,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:54:51,918 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:51,918 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:54:51,918 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:54:51,919 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:51,919 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:51,919 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:51,919 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-15 02:54:51,919 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-15 02:54:51,919 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-15 02:54:51,919 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-15 02:54:51,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-15 02:54:51,919 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-15 02:54:51,919 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-15 02:54:51,919 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:51,919 DEBUG [Thread-1527] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-15 02:54:51,919 DEBUG [Thread-1527] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-15 02:54:51,919 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-15 02:54:51,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-15 02:54:51,919 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-15 02:54:51,928 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=455 (bloomFilter=true), to=hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/f8160290180ba126c904acc32b60eac7/.tmp/A/1f24a205b89340e69eadd12739c7976f 2023-04-15 02:54:52,019 DEBUG [Thread-1527] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-15 02:54:52,020 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-15 02:54:52,021 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-15 02:54:52,022 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-15 02:54:52,022 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-15 02:54:52,022 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-15 02:54:52,022 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-15 02:54:52,022 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-15 02:54:52,022 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-15 02:54:52,022 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:52,022 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:54:52,023 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-15 02:54:52,023 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-15 02:54:52,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-15 02:54:52,023 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:52,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:54:52,024 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:52,024 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:52,024 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-15 02:54:52,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-15 02:54:52,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:54:52,024 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-15 02:54:52,024 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-15 02:54:52,024 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-15 02:54:52,024 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-15 02:54:52,024 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-15 02:54:52,024 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-15 02:54:52,024 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-15 02:54:52,025 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-15 02:54:52,025 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681527274083.f8160290180ba126c904acc32b60eac7. 2023-04-15 02:54:52,025 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681527274083.f8160290180ba126c904acc32b60eac7. started... 2023-04-15 02:54:52,025 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681527274083.f8160290180ba126c904acc32b60eac7. as already flushing 2023-04-15 02:54:52,025 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681527274083.f8160290180ba126c904acc32b60eac7. 2023-04-15 02:54:52,025 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-15 02:54:52,025 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-15 02:54:52,025 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:52,025 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-15 02:54:52,025 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34681,1681527209133' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-15 02:54:52,026 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:52,026 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:52,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:52,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:54:52,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:54:52,026 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:52,026 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-15 02:54:52,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:54:52,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:54:52,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:54:52,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:52,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:54:52,041 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34681,1681527209133' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-15 02:54:52,041 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@54e5cbcc[Count = 0] remaining members to acquire global barrier 2023-04-15 02:54:52,041 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-15 02:54:52,041 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:52,042 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:52,042 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:52,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:52,043 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-15 02:54:52,043 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-15 02:54:52,043 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34681,1681527209133' in zk 2023-04-15 02:54:52,043 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:52,043 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-15 02:54:52,043 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-15 02:54:52,043 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-15 02:54:52,043 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-15 02:54:52,043 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:52,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:52,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:54:52,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:54:52,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:54:52,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:54:52,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:54:52,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:52,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:54:52,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:54:52,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:52,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34681,1681527209133': 2023-04-15 02:54:52,046 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34681,1681527209133' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-15 02:54:52,046 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-15 02:54:52,046 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-15 02:54:52,047 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-15 02:54:52,047 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-15 02:54:52,047 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-15 02:54:52,048 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:52,048 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:52,048 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:52,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:52,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:52,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:54:52,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:54:52,048 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-15 02:54:52,049 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:52,049 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-15 02:54:52,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-15 02:54:52,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:54:52,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:52,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:54:52,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:54:52,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:54:52,050 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:52,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:52,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:54:52,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:54:52,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:52,053 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:52,053 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:54:52,053 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:54:52,053 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:52,053 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-15 02:54:52,053 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-15 02:54:52,053 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-15 02:54:52,053 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-15 02:54:52,053 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-15 02:54:52,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-15 02:54:52,053 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-15 02:54:52,053 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:52,053 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:52,053 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:52,053 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-15 02:54:52,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-15 02:54:52,053 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-15 02:54:52,053 DEBUG [Thread-1527] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-15 02:54:52,053 DEBUG [Thread-1527] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-15 02:54:52,153 DEBUG [Thread-1527] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-15 02:54:52,154 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-15 02:54:52,155 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-15 02:54:52,156 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-15 02:54:52,156 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-15 02:54:52,156 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-15 02:54:52,156 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-15 02:54:52,156 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-15 02:54:52,156 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-15 02:54:52,156 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:52,156 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:54:52,157 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:52,157 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-15 02:54:52,157 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-15 02:54:52,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-15 02:54:52,158 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:52,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:54:52,158 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-15 02:54:52,158 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:52,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-15 02:54:52,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:54:52,158 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-15 02:54:52,159 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-15 02:54:52,159 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-15 02:54:52,159 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-15 02:54:52,159 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-15 02:54:52,159 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-15 02:54:52,159 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-15 02:54:52,159 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-15 02:54:52,159 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681527274083.f8160290180ba126c904acc32b60eac7. 2023-04-15 02:54:52,159 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681527274083.f8160290180ba126c904acc32b60eac7. started... 2023-04-15 02:54:52,159 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681527274083.f8160290180ba126c904acc32b60eac7. as already flushing 2023-04-15 02:54:52,159 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681527274083.f8160290180ba126c904acc32b60eac7. 2023-04-15 02:54:52,159 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-15 02:54:52,159 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-15 02:54:52,159 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:52,159 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-15 02:54:52,160 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34681,1681527209133' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-15 02:54:52,160 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:52,160 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:52,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:52,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:54:52,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:54:52,161 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:52,161 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-15 02:54:52,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:54:52,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:54:52,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:54:52,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:52,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:54:52,162 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34681,1681527209133' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-15 02:54:52,162 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7b9659e2[Count = 0] remaining members to acquire global barrier 2023-04-15 02:54:52,162 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-15 02:54:52,162 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:52,163 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:52,163 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:52,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:52,163 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-15 02:54:52,163 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-15 02:54:52,163 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:52,163 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-15 02:54:52,163 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34681,1681527209133' in zk 2023-04-15 02:54:52,164 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:52,164 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-15 02:54:52,164 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-15 02:54:52,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:52,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:54:52,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:54:52,164 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-15 02:54:52,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:54:52,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:54:52,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:54:52,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:52,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:54:52,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:54:52,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:52,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34681,1681527209133': 2023-04-15 02:54:52,166 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34681,1681527209133' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-15 02:54:52,166 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-15 02:54:52,166 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-15 02:54:52,167 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-15 02:54:52,167 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-15 02:54:52,167 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-15 02:54:52,167 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:52,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:52,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:54:52,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:54:52,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:54:52,168 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:52,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:54:52,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:54:52,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:54:52,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:52,169 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:52,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:54:52,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:54:52,169 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:52,169 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-15 02:54:52,169 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:52,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:52,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:52,170 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-15 02:54:52,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-15 02:54:52,171 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:52,171 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:54:52,171 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:54:52,171 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:52,171 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-15 02:54:52,171 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-15 02:54:52,171 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:52,171 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-15 02:54:52,171 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-15 02:54:52,171 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-15 02:54:52,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-15 02:54:52,171 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:52,171 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:52,171 DEBUG [Thread-1527] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-15 02:54:52,171 DEBUG [Thread-1527] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-15 02:54:52,213 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f8160290180ba126c904acc32b60eac7, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:54:52,214 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34681] ipc.CallRunner(144): callId: 416 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:41918 deadline: 1681527352212, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f8160290180ba126c904acc32b60eac7, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:52,271 DEBUG [Thread-1527] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-15 02:54:52,272 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-15 02:54:52,273 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-15 02:54:52,274 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-15 02:54:52,274 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-15 02:54:52,274 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-15 02:54:52,274 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-15 02:54:52,274 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-15 02:54:52,274 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-15 02:54:52,275 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:52,275 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:54:52,275 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:52,275 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-15 02:54:52,275 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-15 02:54:52,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-15 02:54:52,276 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:52,276 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-15 02:54:52,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:54:52,276 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:52,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-15 02:54:52,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:54:52,276 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-15 02:54:52,276 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-15 02:54:52,276 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-15 02:54:52,277 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-15 02:54:52,277 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-15 02:54:52,277 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-15 02:54:52,277 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-15 02:54:52,277 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-15 02:54:52,277 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681527274083.f8160290180ba126c904acc32b60eac7. 2023-04-15 02:54:52,277 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681527274083.f8160290180ba126c904acc32b60eac7. started... 2023-04-15 02:54:52,277 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681527274083.f8160290180ba126c904acc32b60eac7. as already flushing 2023-04-15 02:54:52,277 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681527274083.f8160290180ba126c904acc32b60eac7. 2023-04-15 02:54:52,277 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-15 02:54:52,277 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-15 02:54:52,277 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:52,277 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-15 02:54:52,277 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34681,1681527209133' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-15 02:54:52,278 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:52,278 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:52,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:52,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:54:52,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:54:52,278 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:52,278 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-15 02:54:52,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:54:52,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:54:52,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:54:52,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:52,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:54:52,279 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34681,1681527209133' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-15 02:54:52,279 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-15 02:54:52,279 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7c72714d[Count = 0] remaining members to acquire global barrier 2023-04-15 02:54:52,279 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:52,280 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:52,280 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:52,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:52,280 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-15 02:54:52,280 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-15 02:54:52,280 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34681,1681527209133' in zk 2023-04-15 02:54:52,280 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:52,280 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-15 02:54:52,281 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:52,281 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-15 02:54:52,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:52,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:54:52,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:54:52,281 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-15 02:54:52,281 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-15 02:54:52,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:54:52,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:54:52,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:54:52,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:52,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:54:52,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:54:52,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:52,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34681,1681527209133': 2023-04-15 02:54:52,283 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34681,1681527209133' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-15 02:54:52,283 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-15 02:54:52,283 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-15 02:54:52,284 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-15 02:54:52,284 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-15 02:54:52,284 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-15 02:54:52,284 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:52,284 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:52,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:52,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:54:52,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:54:52,285 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:52,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:52,285 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-15 02:54:52,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:54:52,285 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-15 02:54:52,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-15 02:54:52,286 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:52,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:52,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:54:52,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:54:52,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:54:52,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:52,288 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:52,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:54:52,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:54:52,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:52,295 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:52,295 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-15 02:54:52,295 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-15 02:54:52,295 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:54:52,295 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-15 02:54:52,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-15 02:54:52,295 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-15 02:54:52,295 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-15 02:54:52,295 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-15 02:54:52,295 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:54:52,295 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:52,295 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:52,295 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:52,295 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-15 02:54:52,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-15 02:54:52,295 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:52,295 DEBUG [Thread-1527] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-15 02:54:52,296 DEBUG [Thread-1527] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-15 02:54:52,339 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=455 (bloomFilter=true), to=hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/f8160290180ba126c904acc32b60eac7/.tmp/B/ee48283915b74aa6b528d9fb9f3189f7 2023-04-15 02:54:52,396 DEBUG [Thread-1527] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-15 02:54:52,396 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-15 02:54:52,398 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-15 02:54:52,399 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-15 02:54:52,399 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-15 02:54:52,399 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-15 02:54:52,399 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-15 02:54:52,399 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-15 02:54:52,399 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-15 02:54:52,399 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:52,399 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:54:52,400 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:52,400 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-15 02:54:52,400 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-15 02:54:52,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-15 02:54:52,400 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:52,400 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-15 02:54:52,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:54:52,401 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:52,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-15 02:54:52,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:54:52,401 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-15 02:54:52,401 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-15 02:54:52,401 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-15 02:54:52,401 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-15 02:54:52,401 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-15 02:54:52,401 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-15 02:54:52,402 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-15 02:54:52,402 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-15 02:54:52,402 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681527274083.f8160290180ba126c904acc32b60eac7. 2023-04-15 02:54:52,402 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681527274083.f8160290180ba126c904acc32b60eac7. started... 2023-04-15 02:54:52,402 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681527274083.f8160290180ba126c904acc32b60eac7. as already flushing 2023-04-15 02:54:52,402 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681527274083.f8160290180ba126c904acc32b60eac7. 2023-04-15 02:54:52,402 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-15 02:54:52,402 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-15 02:54:52,402 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:52,402 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-15 02:54:52,402 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34681,1681527209133' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-15 02:54:52,403 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:52,403 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:52,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:52,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:54:52,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:54:52,403 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:52,403 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-15 02:54:52,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:54:52,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:54:52,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:54:52,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:52,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:54:52,404 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34681,1681527209133' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-15 02:54:52,404 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6dd5d509[Count = 0] remaining members to acquire global barrier 2023-04-15 02:54:52,404 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-15 02:54:52,404 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:52,405 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:52,405 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:52,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:52,405 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:52,405 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-15 02:54:52,405 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-15 02:54:52,405 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-15 02:54:52,405 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34681,1681527209133' in zk 2023-04-15 02:54:52,405 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:52,405 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-15 02:54:52,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:52,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:54:52,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:54:52,405 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-15 02:54:52,406 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-15 02:54:52,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:54:52,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:54:52,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:54:52,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:52,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:54:52,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:54:52,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:52,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34681,1681527209133': 2023-04-15 02:54:52,407 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34681,1681527209133' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-15 02:54:52,407 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-15 02:54:52,407 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-15 02:54:52,407 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-15 02:54:52,407 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-15 02:54:52,407 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-15 02:54:52,408 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:52,408 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:52,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:52,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:54:52,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:54:52,408 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-15 02:54:52,408 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:52,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:52,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:54:52,408 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:52,408 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-15 02:54:52,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-15 02:54:52,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:54:52,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:52,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:54:52,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:54:52,409 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:52,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:52,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:54:52,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:54:52,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:52,410 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:52,410 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-15 02:54:52,410 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:54:52,410 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-15 02:54:52,410 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:54:52,410 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:52,410 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-15 02:54:52,410 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-15 02:54:52,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-15 02:54:52,410 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-15 02:54:52,411 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-15 02:54:52,410 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:52,411 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:52,411 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:52,410 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-15 02:54:52,411 DEBUG [Thread-1527] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-15 02:54:52,411 DEBUG [Thread-1527] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-15 02:54:52,411 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-15 02:54:52,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-15 02:54:52,511 DEBUG [Thread-1527] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-15 02:54:52,513 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-15 02:54:52,517 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-15 02:54:52,518 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-15 02:54:52,518 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-15 02:54:52,519 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-15 02:54:52,519 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-15 02:54:52,519 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-15 02:54:52,519 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-15 02:54:52,519 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:52,519 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:54:52,522 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-15 02:54:52,522 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:52,522 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-15 02:54:52,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-15 02:54:52,522 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:52,522 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-15 02:54:52,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:54:52,523 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:52,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-15 02:54:52,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:54:52,523 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-15 02:54:52,526 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-15 02:54:52,526 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-15 02:54:52,526 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-15 02:54:52,526 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-15 02:54:52,526 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-15 02:54:52,529 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-15 02:54:52,529 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-15 02:54:52,529 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681527274083.f8160290180ba126c904acc32b60eac7. 2023-04-15 02:54:52,529 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681527274083.f8160290180ba126c904acc32b60eac7. started... 2023-04-15 02:54:52,529 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681527274083.f8160290180ba126c904acc32b60eac7. as already flushing 2023-04-15 02:54:52,530 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681527274083.f8160290180ba126c904acc32b60eac7. 2023-04-15 02:54:52,530 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-15 02:54:52,530 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-15 02:54:52,530 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:52,530 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-15 02:54:52,530 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34681,1681527209133' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-15 02:54:52,533 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:52,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:52,533 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:52,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:54:52,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:54:52,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:54:52,533 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:52,533 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-15 02:54:52,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:54:52,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:54:52,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:52,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:54:52,535 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34681,1681527209133' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-15 02:54:52,535 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1714c8d4[Count = 0] remaining members to acquire global barrier 2023-04-15 02:54:52,535 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-15 02:54:52,535 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:52,536 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:52,536 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:52,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:52,536 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-15 02:54:52,536 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-15 02:54:52,536 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34681,1681527209133' in zk 2023-04-15 02:54:52,537 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:52,537 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-15 02:54:52,537 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:52,537 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-15 02:54:52,537 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-15 02:54:52,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:52,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:54:52,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:54:52,537 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-15 02:54:52,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:54:52,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:54:52,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:54:52,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:52,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:54:52,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:54:52,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:52,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34681,1681527209133': 2023-04-15 02:54:52,540 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34681,1681527209133' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-15 02:54:52,540 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-15 02:54:52,540 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-15 02:54:52,540 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-15 02:54:52,540 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-15 02:54:52,540 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-15 02:54:52,541 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:52,541 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:52,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:52,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:54:52,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:54:52,541 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:52,541 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-15 02:54:52,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:52,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:54:52,542 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:52,542 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-15 02:54:52,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-15 02:54:52,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:54:52,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:52,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:54:52,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:54:52,543 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:52,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:52,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:54:52,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:54:52,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:52,544 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:52,544 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:54:52,544 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:54:52,544 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-15 02:54:52,544 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-15 02:54:52,544 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-15 02:54:52,544 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:52,544 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-15 02:54:52,544 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-15 02:54:52,544 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:52,544 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-15 02:54:52,544 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-15 02:54:52,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-15 02:54:52,544 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:52,544 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:52,544 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-15 02:54:52,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-15 02:54:52,545 DEBUG [Thread-1527] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-15 02:54:52,545 DEBUG [Thread-1527] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-15 02:54:52,645 DEBUG [Thread-1527] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-15 02:54:52,645 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-15 02:54:52,647 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-15 02:54:52,647 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-15 02:54:52,647 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-15 02:54:52,647 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-15 02:54:52,648 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-15 02:54:52,648 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-15 02:54:52,648 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-15 02:54:52,648 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:52,648 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:54:52,649 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:52,649 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-15 02:54:52,649 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-15 02:54:52,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-15 02:54:52,649 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:52,649 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-15 02:54:52,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:54:52,649 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:52,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-15 02:54:52,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:54:52,650 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-15 02:54:52,650 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-15 02:54:52,650 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-15 02:54:52,650 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-15 02:54:52,650 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-15 02:54:52,650 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-15 02:54:52,651 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-15 02:54:52,651 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-15 02:54:52,651 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681527274083.f8160290180ba126c904acc32b60eac7. 2023-04-15 02:54:52,651 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681527274083.f8160290180ba126c904acc32b60eac7. started... 2023-04-15 02:54:52,651 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681527274083.f8160290180ba126c904acc32b60eac7. as already flushing 2023-04-15 02:54:52,651 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681527274083.f8160290180ba126c904acc32b60eac7. 2023-04-15 02:54:52,651 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-15 02:54:52,651 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-15 02:54:52,651 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:52,651 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-15 02:54:52,651 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34681,1681527209133' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-15 02:54:52,652 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:52,652 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:52,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:52,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:54:52,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:54:52,652 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:52,652 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-15 02:54:52,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:54:52,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:54:52,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:54:52,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:52,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:54:52,653 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34681,1681527209133' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-15 02:54:52,653 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5e84da9d[Count = 0] remaining members to acquire global barrier 2023-04-15 02:54:52,653 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-15 02:54:52,653 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:52,654 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:52,654 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:52,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:52,654 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-15 02:54:52,654 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-15 02:54:52,654 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34681,1681527209133' in zk 2023-04-15 02:54:52,654 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:52,654 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-15 02:54:52,655 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:52,655 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-15 02:54:52,655 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-15 02:54:52,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:52,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:54:52,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:54:52,655 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-15 02:54:52,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:54:52,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:54:52,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:54:52,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:52,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:54:52,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:54:52,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:52,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34681,1681527209133': 2023-04-15 02:54:52,656 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34681,1681527209133' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-15 02:54:52,656 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-15 02:54:52,656 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-15 02:54:52,656 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-15 02:54:52,656 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-15 02:54:52,656 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-15 02:54:52,657 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:52,657 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:52,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:52,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:54:52,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:54:52,657 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-15 02:54:52,657 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:52,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:52,658 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:52,658 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-15 02:54:52,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:54:52,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-15 02:54:52,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:54:52,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:52,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:54:52,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:54:52,659 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:52,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:52,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:54:52,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:54:52,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:52,660 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-15 02:54:52,660 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:52,660 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-15 02:54:52,660 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:54:52,660 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-15 02:54:52,660 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:54:52,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-15 02:54:52,660 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:52,660 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-15 02:54:52,660 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-15 02:54:52,660 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:52,660 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:52,660 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:52,660 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-15 02:54:52,660 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-15 02:54:52,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-15 02:54:52,660 DEBUG [Thread-1527] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-15 02:54:52,661 DEBUG [Thread-1527] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-15 02:54:52,663 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-15 02:54:52,717 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f8160290180ba126c904acc32b60eac7, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:54:52,717 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34681] ipc.CallRunner(144): callId: 418 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:41918 deadline: 1681527352714, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f8160290180ba126c904acc32b60eac7, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:52,750 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=455 (bloomFilter=true), to=hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/f8160290180ba126c904acc32b60eac7/.tmp/C/f307fe7e6a7e487e87ff809130c5f14e 2023-04-15 02:54:52,753 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/f8160290180ba126c904acc32b60eac7/.tmp/A/1f24a205b89340e69eadd12739c7976f as hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/f8160290180ba126c904acc32b60eac7/A/1f24a205b89340e69eadd12739c7976f 2023-04-15 02:54:52,756 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/f8160290180ba126c904acc32b60eac7/A/1f24a205b89340e69eadd12739c7976f, entries=100, sequenceid=455, filesize=9.5 K 2023-04-15 02:54:52,757 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/f8160290180ba126c904acc32b60eac7/.tmp/B/ee48283915b74aa6b528d9fb9f3189f7 as hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/f8160290180ba126c904acc32b60eac7/B/ee48283915b74aa6b528d9fb9f3189f7 2023-04-15 02:54:52,760 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/f8160290180ba126c904acc32b60eac7/B/ee48283915b74aa6b528d9fb9f3189f7, entries=100, sequenceid=455, filesize=9.5 K 2023-04-15 02:54:52,761 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/f8160290180ba126c904acc32b60eac7/.tmp/C/f307fe7e6a7e487e87ff809130c5f14e as hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/f8160290180ba126c904acc32b60eac7/C/f307fe7e6a7e487e87ff809130c5f14e 2023-04-15 02:54:52,761 DEBUG [Thread-1527] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-15 02:54:52,761 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-15 02:54:52,762 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-15 02:54:52,763 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-15 02:54:52,763 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-15 02:54:52,763 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-15 02:54:52,763 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-15 02:54:52,763 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-15 02:54:52,763 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-15 02:54:52,764 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/f8160290180ba126c904acc32b60eac7/C/f307fe7e6a7e487e87ff809130c5f14e, entries=100, sequenceid=455, filesize=9.5 K 2023-04-15 02:54:52,765 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=134.18 KB/137400 for f8160290180ba126c904acc32b60eac7 in 1249ms, sequenceid=455, compaction requested=true 2023-04-15 02:54:52,765 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for f8160290180ba126c904acc32b60eac7: 2023-04-15 02:54:52,765 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-15 02:54:52,765 DEBUG [RS:0;jenkins-hbase20:34681-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-15 02:54:52,765 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-15 02:54:52,766 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=2), splitQueue=0 2023-04-15 02:54:52,766 DEBUG [RS:0;jenkins-hbase20:34681-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-15 02:54:52,766 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:52,766 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:54:52,767 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:52,767 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:52,767 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-15 02:54:52,767 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-15 02:54:52,768 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-15 02:54:52,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-15 02:54:52,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:54:52,768 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:52,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-15 02:54:52,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:54:52,768 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-15 02:54:52,768 DEBUG [RS:0;jenkins-hbase20:34681-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 34762 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-15 02:54:52,768 DEBUG [RS:0;jenkins-hbase20:34681-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 34762 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-15 02:54:52,768 DEBUG [RS:0;jenkins-hbase20:34681-shortCompactions-0] regionserver.HStore(1912): f8160290180ba126c904acc32b60eac7/A is initiating minor compaction (all files) 2023-04-15 02:54:52,769 DEBUG [RS:0;jenkins-hbase20:34681-longCompactions-0] regionserver.HStore(1912): f8160290180ba126c904acc32b60eac7/B is initiating minor compaction (all files) 2023-04-15 02:54:52,769 INFO [RS:0;jenkins-hbase20:34681-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of f8160290180ba126c904acc32b60eac7/A in TestAcidGuarantees,,1681527274083.f8160290180ba126c904acc32b60eac7. 2023-04-15 02:54:52,769 INFO [RS:0;jenkins-hbase20:34681-longCompactions-0] regionserver.HRegion(2259): Starting compaction of f8160290180ba126c904acc32b60eac7/B in TestAcidGuarantees,,1681527274083.f8160290180ba126c904acc32b60eac7. 2023-04-15 02:54:52,769 INFO [RS:0;jenkins-hbase20:34681-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/f8160290180ba126c904acc32b60eac7/A/ed9f1352085e4cd4a6046951a999c3fb, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/f8160290180ba126c904acc32b60eac7/A/88d467e58ea44e1fbf4749826d79cc70, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/f8160290180ba126c904acc32b60eac7/A/1f24a205b89340e69eadd12739c7976f] into tmpdir=hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/f8160290180ba126c904acc32b60eac7/.tmp, totalSize=33.9 K 2023-04-15 02:54:52,769 INFO [RS:0;jenkins-hbase20:34681-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/f8160290180ba126c904acc32b60eac7/B/a87b02c9f45148c6b5788737b0ebb158, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/f8160290180ba126c904acc32b60eac7/B/ea1cf09ab444475b96e383a56dc20362, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/f8160290180ba126c904acc32b60eac7/B/ee48283915b74aa6b528d9fb9f3189f7] into tmpdir=hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/f8160290180ba126c904acc32b60eac7/.tmp, totalSize=33.9 K 2023-04-15 02:54:52,769 DEBUG [RS:0;jenkins-hbase20:34681-shortCompactions-0] compactions.Compactor(207): Compacting ed9f1352085e4cd4a6046951a999c3fb, keycount=150, bloomtype=ROW, size=12.5 K, encoding=NONE, compression=NONE, seqNum=431, earliestPutTs=1681527289235 2023-04-15 02:54:52,769 DEBUG [RS:0;jenkins-hbase20:34681-longCompactions-0] compactions.Compactor(207): Compacting a87b02c9f45148c6b5788737b0ebb158, keycount=150, bloomtype=ROW, size=12.5 K, encoding=NONE, compression=NONE, seqNum=431, earliestPutTs=1681527289235 2023-04-15 02:54:52,769 DEBUG [RS:0;jenkins-hbase20:34681-shortCompactions-0] compactions.Compactor(207): Compacting 88d467e58ea44e1fbf4749826d79cc70, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=440, earliestPutTs=1681527289358 2023-04-15 02:54:52,769 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-15 02:54:52,769 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-15 02:54:52,769 DEBUG [RS:0;jenkins-hbase20:34681-longCompactions-0] compactions.Compactor(207): Compacting ea1cf09ab444475b96e383a56dc20362, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=440, earliestPutTs=1681527289358 2023-04-15 02:54:52,769 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-15 02:54:52,769 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-15 02:54:52,769 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-15 02:54:52,770 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-15 02:54:52,770 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-15 02:54:52,770 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681527274083.f8160290180ba126c904acc32b60eac7. 2023-04-15 02:54:52,770 DEBUG [RS:0;jenkins-hbase20:34681-longCompactions-0] compactions.Compactor(207): Compacting ee48283915b74aa6b528d9fb9f3189f7, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=455, earliestPutTs=1681527291513 2023-04-15 02:54:52,770 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681527274083.f8160290180ba126c904acc32b60eac7. started... 2023-04-15 02:54:52,770 DEBUG [RS:0;jenkins-hbase20:34681-shortCompactions-0] compactions.Compactor(207): Compacting 1f24a205b89340e69eadd12739c7976f, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=455, earliestPutTs=1681527291513 2023-04-15 02:54:52,770 INFO [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing f8160290180ba126c904acc32b60eac7 3/3 column families, dataSize=134.18 KB heapSize=384.98 KB 2023-04-15 02:54:52,770 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK f8160290180ba126c904acc32b60eac7, store=A 2023-04-15 02:54:52,770 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-15 02:54:52,770 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK f8160290180ba126c904acc32b60eac7, store=B 2023-04-15 02:54:52,770 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-15 02:54:52,770 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK f8160290180ba126c904acc32b60eac7, store=C 2023-04-15 02:54:52,770 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-15 02:54:52,775 INFO [RS:0;jenkins-hbase20:34681-longCompactions-0] throttle.PressureAwareThroughputController(145): f8160290180ba126c904acc32b60eac7#B#compaction#246 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-15 02:54:52,776 INFO [RS:0;jenkins-hbase20:34681-shortCompactions-0] throttle.PressureAwareThroughputController(145): f8160290180ba126c904acc32b60eac7#A#compaction#247 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-15 02:54:52,781 INFO [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=492 (bloomFilter=true), to=hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/f8160290180ba126c904acc32b60eac7/.tmp/A/465c74a754914f38b43b1f7bbea09d26 2023-04-15 02:54:53,187 DEBUG [RS:0;jenkins-hbase20:34681-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/f8160290180ba126c904acc32b60eac7/.tmp/B/a9b63f57015d4c7ba5ccf3f0792317a5 as hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/f8160290180ba126c904acc32b60eac7/B/a9b63f57015d4c7ba5ccf3f0792317a5 2023-04-15 02:54:53,193 INFO [RS:0;jenkins-hbase20:34681-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in f8160290180ba126c904acc32b60eac7/B of f8160290180ba126c904acc32b60eac7 into a9b63f57015d4c7ba5ccf3f0792317a5(size=12.6 K), total size for store is 12.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-15 02:54:53,193 DEBUG [RS:0;jenkins-hbase20:34681-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for f8160290180ba126c904acc32b60eac7: 2023-04-15 02:54:53,193 INFO [RS:0;jenkins-hbase20:34681-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681527274083.f8160290180ba126c904acc32b60eac7., storeName=f8160290180ba126c904acc32b60eac7/B, priority=13, startTime=1681527292765; duration=0sec 2023-04-15 02:54:53,193 DEBUG [RS:0;jenkins-hbase20:34681-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-15 02:54:53,193 DEBUG [RS:0;jenkins-hbase20:34681-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-15 02:54:53,194 INFO [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=492 (bloomFilter=true), to=hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/f8160290180ba126c904acc32b60eac7/.tmp/B/219e1fe89c37434394f36af8af7460ec 2023-04-15 02:54:53,195 DEBUG [RS:0;jenkins-hbase20:34681-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 34762 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-15 02:54:53,195 DEBUG [RS:0;jenkins-hbase20:34681-longCompactions-0] regionserver.HStore(1912): f8160290180ba126c904acc32b60eac7/C is initiating minor compaction (all files) 2023-04-15 02:54:53,195 INFO [RS:0;jenkins-hbase20:34681-longCompactions-0] regionserver.HRegion(2259): Starting compaction of f8160290180ba126c904acc32b60eac7/C in TestAcidGuarantees,,1681527274083.f8160290180ba126c904acc32b60eac7. 2023-04-15 02:54:53,195 INFO [RS:0;jenkins-hbase20:34681-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/f8160290180ba126c904acc32b60eac7/C/40a96079c22f4d509d57c890e94b1486, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/f8160290180ba126c904acc32b60eac7/C/0449ab308b884ddf8b41d677a28aff3f, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/f8160290180ba126c904acc32b60eac7/C/f307fe7e6a7e487e87ff809130c5f14e] into tmpdir=hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/f8160290180ba126c904acc32b60eac7/.tmp, totalSize=33.9 K 2023-04-15 02:54:53,196 DEBUG [RS:0;jenkins-hbase20:34681-longCompactions-0] compactions.Compactor(207): Compacting 40a96079c22f4d509d57c890e94b1486, keycount=150, bloomtype=ROW, size=12.5 K, encoding=NONE, compression=NONE, seqNum=431, earliestPutTs=1681527289235 2023-04-15 02:54:53,196 DEBUG [RS:0;jenkins-hbase20:34681-longCompactions-0] compactions.Compactor(207): Compacting 0449ab308b884ddf8b41d677a28aff3f, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=440, earliestPutTs=1681527289358 2023-04-15 02:54:53,196 DEBUG [RS:0;jenkins-hbase20:34681-longCompactions-0] compactions.Compactor(207): Compacting f307fe7e6a7e487e87ff809130c5f14e, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=455, earliestPutTs=1681527291513 2023-04-15 02:54:53,200 DEBUG [RS:0;jenkins-hbase20:34681-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/f8160290180ba126c904acc32b60eac7/.tmp/A/9004437d3cc24f318e2e9d2d5d21a388 as hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/f8160290180ba126c904acc32b60eac7/A/9004437d3cc24f318e2e9d2d5d21a388 2023-04-15 02:54:53,211 INFO [RS:0;jenkins-hbase20:34681-longCompactions-0] throttle.PressureAwareThroughputController(145): f8160290180ba126c904acc32b60eac7#C#compaction#249 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-15 02:54:53,215 INFO [RS:0;jenkins-hbase20:34681-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in f8160290180ba126c904acc32b60eac7/A of f8160290180ba126c904acc32b60eac7 into 9004437d3cc24f318e2e9d2d5d21a388(size=12.6 K), total size for store is 12.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-15 02:54:53,216 DEBUG [RS:0;jenkins-hbase20:34681-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for f8160290180ba126c904acc32b60eac7: 2023-04-15 02:54:53,216 INFO [RS:0;jenkins-hbase20:34681-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681527274083.f8160290180ba126c904acc32b60eac7., storeName=f8160290180ba126c904acc32b60eac7/A, priority=13, startTime=1681527292765; duration=0sec 2023-04-15 02:54:53,216 DEBUG [RS:0;jenkins-hbase20:34681-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-15 02:54:53,660 DEBUG [RS:0;jenkins-hbase20:34681-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/f8160290180ba126c904acc32b60eac7/.tmp/C/6f5b2fbcc01d4e2eb50f02d4ead85ce8 as hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/f8160290180ba126c904acc32b60eac7/C/6f5b2fbcc01d4e2eb50f02d4ead85ce8 2023-04-15 02:54:53,661 INFO [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=492 (bloomFilter=true), to=hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/f8160290180ba126c904acc32b60eac7/.tmp/C/ca8ca18f22304064843cb86197b6fb9f 2023-04-15 02:54:53,664 INFO [RS:0;jenkins-hbase20:34681-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in f8160290180ba126c904acc32b60eac7/C of f8160290180ba126c904acc32b60eac7 into 6f5b2fbcc01d4e2eb50f02d4ead85ce8(size=12.6 K), total size for store is 12.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-15 02:54:53,665 DEBUG [RS:0;jenkins-hbase20:34681-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for f8160290180ba126c904acc32b60eac7: 2023-04-15 02:54:53,665 INFO [RS:0;jenkins-hbase20:34681-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681527274083.f8160290180ba126c904acc32b60eac7., storeName=f8160290180ba126c904acc32b60eac7/C, priority=13, startTime=1681527292766; duration=0sec 2023-04-15 02:54:53,665 DEBUG [RS:0;jenkins-hbase20:34681-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-15 02:54:53,665 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/f8160290180ba126c904acc32b60eac7/.tmp/A/465c74a754914f38b43b1f7bbea09d26 as hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/f8160290180ba126c904acc32b60eac7/A/465c74a754914f38b43b1f7bbea09d26 2023-04-15 02:54:53,668 INFO [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/f8160290180ba126c904acc32b60eac7/A/465c74a754914f38b43b1f7bbea09d26, entries=150, sequenceid=492, filesize=11.9 K 2023-04-15 02:54:53,669 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/f8160290180ba126c904acc32b60eac7/.tmp/B/219e1fe89c37434394f36af8af7460ec as hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/f8160290180ba126c904acc32b60eac7/B/219e1fe89c37434394f36af8af7460ec 2023-04-15 02:54:53,673 INFO [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/f8160290180ba126c904acc32b60eac7/B/219e1fe89c37434394f36af8af7460ec, entries=150, sequenceid=492, filesize=11.9 K 2023-04-15 02:54:53,675 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/f8160290180ba126c904acc32b60eac7/.tmp/C/ca8ca18f22304064843cb86197b6fb9f as hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/f8160290180ba126c904acc32b60eac7/C/ca8ca18f22304064843cb86197b6fb9f 2023-04-15 02:54:53,678 INFO [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/f8160290180ba126c904acc32b60eac7/C/ca8ca18f22304064843cb86197b6fb9f, entries=150, sequenceid=492, filesize=11.9 K 2023-04-15 02:54:53,679 INFO [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~134.18 KB/137400, heapSize ~384.94 KB/394176, currentSize=0 B/0 for f8160290180ba126c904acc32b60eac7 in 909ms, sequenceid=492, compaction requested=false 2023-04-15 02:54:53,679 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for f8160290180ba126c904acc32b60eac7: 2023-04-15 02:54:53,679 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681527274083.f8160290180ba126c904acc32b60eac7. 2023-04-15 02:54:53,679 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-15 02:54:53,679 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-15 02:54:53,679 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:53,679 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-15 02:54:53,679 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34681,1681527209133' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-15 02:54:53,680 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:53,680 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:53,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:53,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:54:53,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:54:53,680 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:53,680 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-15 02:54:53,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:54:53,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:54:53,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:54:53,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:53,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:54:53,681 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34681,1681527209133' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-15 02:54:53,682 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@31132019[Count = 0] remaining members to acquire global barrier 2023-04-15 02:54:53,682 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-15 02:54:53,682 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:53,682 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:53,682 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:53,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:53,682 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-15 02:54:53,682 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-15 02:54:53,682 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34681,1681527209133' in zk 2023-04-15 02:54:53,683 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:53,683 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-15 02:54:53,683 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-15 02:54:53,683 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-15 02:54:53,683 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-15 02:54:53,683 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:53,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:53,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:54:53,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:54:53,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:54:53,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:54:53,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:54:53,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:53,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:54:53,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:54:53,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:53,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34681,1681527209133': 2023-04-15 02:54:53,686 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34681,1681527209133' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-15 02:54:53,686 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-15 02:54:53,686 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-15 02:54:53,686 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-15 02:54:53,686 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-15 02:54:53,686 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-15 02:54:53,687 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:53,687 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-15 02:54:53,687 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:53,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:53,687 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:53,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:53,688 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-15 02:54:53,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-15 02:54:53,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:54:53,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:54:53,688 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:53,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:53,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:54:53,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:54:53,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:54:53,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:54:53,689 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:53,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:53,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:54:53,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:54:53,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:53,690 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-15 02:54:53,690 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-15 02:54:53,690 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-15 02:54:53,690 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-15 02:54:53,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-15 02:54:53,690 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-15 02:54:53,691 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-15 02:54:53,691 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:53,691 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:54:53,691 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:54:53,691 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:53,691 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:53,691 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:53,691 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-15 02:54:53,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-15 02:54:53,691 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:53,691 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-15 02:54:53,691 DEBUG [Thread-1527] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-15 02:54:53,691 DEBUG [Thread-1527] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-15 02:54:53,736 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34681] regionserver.HRegion(9158): Flush requested on f8160290180ba126c904acc32b60eac7 2023-04-15 02:54:53,736 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing f8160290180ba126c904acc32b60eac7 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-15 02:54:53,737 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK f8160290180ba126c904acc32b60eac7, store=A 2023-04-15 02:54:53,737 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-15 02:54:53,737 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK f8160290180ba126c904acc32b60eac7, store=B 2023-04-15 02:54:53,737 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-15 02:54:53,737 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK f8160290180ba126c904acc32b60eac7, store=C 2023-04-15 02:54:53,737 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-15 02:54:53,778 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681527293757, max timestamp=1681527293776Num uniques 859; 2023-04-15 02:54:53,779 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681527293757, max timestamp=1681527293776Num uniques 859; 2023-04-15 02:54:53,779 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681527293757, max timestamp=1681527293776Num uniques 859; 2023-04-15 02:54:53,779 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-1] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-15 02:54:53,779 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-8] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-15 02:54:53,779 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-7] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-15 02:54:53,791 DEBUG [Thread-1527] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-15 02:54:53,792 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-15 02:54:53,793 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-15 02:54:53,795 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-15 02:54:53,795 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-15 02:54:53,795 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-15 02:54:53,795 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-15 02:54:53,795 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-15 02:54:53,795 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-15 02:54:53,795 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:53,795 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:54:53,796 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-15 02:54:53,796 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-15 02:54:53,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-15 02:54:53,796 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:53,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:54:53,797 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:53,797 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-15 02:54:53,797 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:53,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-15 02:54:53,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:54:53,797 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-15 02:54:53,797 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-15 02:54:53,798 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-15 02:54:53,798 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-15 02:54:53,798 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-15 02:54:53,798 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-15 02:54:53,798 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-15 02:54:53,798 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681527274083.f8160290180ba126c904acc32b60eac7. 2023-04-15 02:54:53,798 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-15 02:54:53,798 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681527274083.f8160290180ba126c904acc32b60eac7. started... 2023-04-15 02:54:53,798 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681527274083.f8160290180ba126c904acc32b60eac7. as already flushing 2023-04-15 02:54:53,798 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681527274083.f8160290180ba126c904acc32b60eac7. 2023-04-15 02:54:53,798 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-15 02:54:53,798 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-15 02:54:53,798 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:53,798 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-15 02:54:53,798 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34681,1681527209133' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-15 02:54:53,800 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:53,800 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:53,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:53,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:54:53,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:54:53,801 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:53,801 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-15 02:54:53,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:54:53,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:54:53,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:54:53,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:53,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:54:53,802 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34681,1681527209133' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-15 02:54:53,802 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@72add6af[Count = 0] remaining members to acquire global barrier 2023-04-15 02:54:53,802 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-15 02:54:53,802 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:53,803 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:53,803 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:53,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:53,803 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-15 02:54:53,803 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-15 02:54:53,803 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34681,1681527209133' in zk 2023-04-15 02:54:53,803 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:53,804 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-15 02:54:53,804 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-15 02:54:53,804 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-15 02:54:53,804 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:53,804 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-15 02:54:53,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:53,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:54:53,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:54:53,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:54:53,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:54:53,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:54:53,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:53,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:54:53,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:54:53,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:53,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34681,1681527209133': 2023-04-15 02:54:53,807 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34681,1681527209133' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-15 02:54:53,807 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-15 02:54:53,807 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-15 02:54:53,807 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-15 02:54:53,807 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-15 02:54:53,807 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-15 02:54:53,808 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:53,808 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:53,808 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:53,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:53,808 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-15 02:54:53,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:53,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:54:53,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:54:53,809 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-15 02:54:53,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-15 02:54:53,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:54:53,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:53,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:54:53,810 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:53,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:54:53,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:54:53,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:53,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:54:53,811 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:53,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:54:53,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:53,812 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:53,812 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:54:53,812 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:54:53,812 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-15 02:54:53,812 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-15 02:54:53,812 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-15 02:54:53,813 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-15 02:54:53,812 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:53,813 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:53,813 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:53,813 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:53,813 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-15 02:54:53,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-15 02:54:53,813 DEBUG [Thread-1527] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-15 02:54:53,813 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-15 02:54:53,813 DEBUG [Thread-1527] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-15 02:54:53,813 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-15 02:54:53,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-15 02:54:53,831 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f8160290180ba126c904acc32b60eac7, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:54:53,831 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34681] ipc.CallRunner(144): callId: 461 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:41918 deadline: 1681527353829, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f8160290180ba126c904acc32b60eac7, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:53,913 DEBUG [Thread-1527] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-15 02:54:53,914 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-15 02:54:53,915 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-15 02:54:53,916 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-15 02:54:53,916 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-15 02:54:53,916 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-15 02:54:53,916 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-15 02:54:53,916 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-15 02:54:53,916 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-15 02:54:53,916 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:53,917 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:54:53,917 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-15 02:54:53,917 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-15 02:54:53,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-15 02:54:53,917 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:53,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:54:53,918 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:53,918 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-15 02:54:53,918 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:53,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-15 02:54:53,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:54:53,918 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-15 02:54:53,919 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-15 02:54:53,919 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-15 02:54:53,919 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-15 02:54:53,919 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-15 02:54:53,919 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-15 02:54:53,920 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-15 02:54:53,920 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-15 02:54:53,920 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681527274083.f8160290180ba126c904acc32b60eac7. 2023-04-15 02:54:53,921 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681527274083.f8160290180ba126c904acc32b60eac7. started... 2023-04-15 02:54:53,921 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681527274083.f8160290180ba126c904acc32b60eac7. as already flushing 2023-04-15 02:54:53,921 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681527274083.f8160290180ba126c904acc32b60eac7. 2023-04-15 02:54:53,921 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-15 02:54:53,921 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-15 02:54:53,921 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:53,921 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-15 02:54:53,921 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34681,1681527209133' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-15 02:54:53,922 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:53,922 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:53,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:53,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:54:53,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:54:53,922 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:53,922 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-15 02:54:53,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:54:53,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:54:53,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:54:53,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:53,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:54:53,923 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34681,1681527209133' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-15 02:54:53,923 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-15 02:54:53,923 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@434ee246[Count = 0] remaining members to acquire global barrier 2023-04-15 02:54:53,923 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:53,924 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:53,924 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:53,924 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-15 02:54:53,924 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:53,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:53,924 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-15 02:54:53,924 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-15 02:54:53,924 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34681,1681527209133' in zk 2023-04-15 02:54:53,925 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:53,925 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-15 02:54:53,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:53,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:54:53,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:54:53,925 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-15 02:54:53,926 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-15 02:54:53,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:54:53,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:54:53,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:54:53,934 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f8160290180ba126c904acc32b60eac7, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:54:53,934 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34681] ipc.CallRunner(144): callId: 463 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:41918 deadline: 1681527353932, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f8160290180ba126c904acc32b60eac7, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:53,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:53,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:54:53,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:54:53,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:53,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34681,1681527209133': 2023-04-15 02:54:53,948 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34681,1681527209133' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-15 02:54:53,948 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-15 02:54:53,948 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-15 02:54:53,948 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-15 02:54:53,948 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-15 02:54:53,948 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-15 02:54:53,949 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:53,949 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:53,949 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-15 02:54:53,949 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:53,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:53,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:53,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:54:53,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:54:53,949 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-15 02:54:53,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-15 02:54:53,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:53,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:54:53,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:54:53,951 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:53,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:54:53,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:54:53,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:53,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:54:53,951 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:53,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:54:53,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:53,953 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:53,953 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:54:53,953 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:54:53,953 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-15 02:54:53,953 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-15 02:54:53,953 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:53,953 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-15 02:54:53,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-15 02:54:53,953 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:53,953 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-15 02:54:53,953 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-15 02:54:53,953 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-15 02:54:53,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-15 02:54:53,954 DEBUG [Thread-1527] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-15 02:54:53,954 DEBUG [Thread-1527] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-15 02:54:53,953 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-15 02:54:53,953 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:53,954 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:54,054 DEBUG [Thread-1527] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-15 02:54:54,054 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-15 02:54:54,055 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-15 02:54:54,056 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-15 02:54:54,056 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-15 02:54:54,056 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-15 02:54:54,057 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-15 02:54:54,057 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-15 02:54:54,057 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-15 02:54:54,057 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:54,057 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:54:54,058 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:54,058 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-15 02:54:54,058 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-15 02:54:54,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-15 02:54:54,058 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:54,058 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-15 02:54:54,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:54:54,059 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:54,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-15 02:54:54,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:54:54,059 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-15 02:54:54,059 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-15 02:54:54,059 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-15 02:54:54,059 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-15 02:54:54,059 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-15 02:54:54,059 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-15 02:54:54,061 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-15 02:54:54,061 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-15 02:54:54,061 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681527274083.f8160290180ba126c904acc32b60eac7. 2023-04-15 02:54:54,061 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681527274083.f8160290180ba126c904acc32b60eac7. started... 2023-04-15 02:54:54,061 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681527274083.f8160290180ba126c904acc32b60eac7. as already flushing 2023-04-15 02:54:54,061 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681527274083.f8160290180ba126c904acc32b60eac7. 2023-04-15 02:54:54,061 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-15 02:54:54,061 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-15 02:54:54,061 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:54,061 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-15 02:54:54,061 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34681,1681527209133' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-15 02:54:54,063 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:54,063 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:54,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:54,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:54:54,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:54:54,063 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:54,063 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-15 02:54:54,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:54:54,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:54:54,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:54:54,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:54,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:54:54,065 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34681,1681527209133' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-15 02:54:54,065 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-15 02:54:54,065 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3be69880[Count = 0] remaining members to acquire global barrier 2023-04-15 02:54:54,065 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:54,065 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:54,066 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:54,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:54,066 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-15 02:54:54,066 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-15 02:54:54,066 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34681,1681527209133' in zk 2023-04-15 02:54:54,066 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:54,066 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-15 02:54:54,066 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:54,066 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-15 02:54:54,066 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-15 02:54:54,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:54,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:54:54,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:54:54,066 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-15 02:54:54,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:54:54,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:54:54,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:54:54,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:54,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:54:54,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:54:54,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:54,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34681,1681527209133': 2023-04-15 02:54:54,068 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34681,1681527209133' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-15 02:54:54,068 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-15 02:54:54,068 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-15 02:54:54,068 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-15 02:54:54,068 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-15 02:54:54,068 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-15 02:54:54,069 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:54,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:54,069 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:54,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:54:54,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:54:54,069 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:54,069 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-15 02:54:54,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:54,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:54:54,070 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:54,070 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-15 02:54:54,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-15 02:54:54,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:54:54,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:54,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:54:54,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:54:54,071 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:54,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:54,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:54:54,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:54:54,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:54,083 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-15 02:54:54,083 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:54,083 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-15 02:54:54,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-15 02:54:54,083 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-15 02:54:54,083 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-15 02:54:54,083 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-15 02:54:54,083 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-15 02:54:54,083 DEBUG [Thread-1527] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-15 02:54:54,083 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:54:54,083 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-15 02:54:54,083 DEBUG [Thread-1527] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-15 02:54:54,083 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-15 02:54:54,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-15 02:54:54,083 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:54:54,084 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:54,084 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:54,084 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:54,084 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:54,139 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f8160290180ba126c904acc32b60eac7, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:54:54,139 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34681] ipc.CallRunner(144): callId: 465 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:41918 deadline: 1681527354136, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f8160290180ba126c904acc32b60eac7, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:54,145 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=505 (bloomFilter=true), to=hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/f8160290180ba126c904acc32b60eac7/.tmp/A/8cc3365625a74bbe9ffca4472e1584e8 2023-04-15 02:54:54,152 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=505 (bloomFilter=true), to=hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/f8160290180ba126c904acc32b60eac7/.tmp/B/805958efea2344fdb254357346ad44de 2023-04-15 02:54:54,184 DEBUG [Thread-1527] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-15 02:54:54,184 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-15 02:54:54,185 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-15 02:54:54,186 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-15 02:54:54,186 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-15 02:54:54,186 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-15 02:54:54,186 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-15 02:54:54,186 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-15 02:54:54,186 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-15 02:54:54,186 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:54,186 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:54:54,187 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-15 02:54:54,187 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:54,187 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-15 02:54:54,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-15 02:54:54,187 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:54,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:54:54,187 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-15 02:54:54,187 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:54,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-15 02:54:54,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:54:54,188 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-15 02:54:54,188 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-15 02:54:54,188 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-15 02:54:54,188 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-15 02:54:54,188 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-15 02:54:54,188 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-15 02:54:54,188 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-15 02:54:54,188 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-15 02:54:54,188 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681527274083.f8160290180ba126c904acc32b60eac7. 2023-04-15 02:54:54,188 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681527274083.f8160290180ba126c904acc32b60eac7. started... 2023-04-15 02:54:54,188 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681527274083.f8160290180ba126c904acc32b60eac7. as already flushing 2023-04-15 02:54:54,189 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681527274083.f8160290180ba126c904acc32b60eac7. 2023-04-15 02:54:54,189 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-15 02:54:54,189 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-15 02:54:54,189 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:54,189 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-15 02:54:54,189 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34681,1681527209133' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-15 02:54:54,189 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:54,189 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:54,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:54,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:54:54,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:54:54,190 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:54,190 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-15 02:54:54,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:54:54,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:54:54,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:54:54,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:54,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:54:54,191 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34681,1681527209133' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-15 02:54:54,191 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@13dd640d[Count = 0] remaining members to acquire global barrier 2023-04-15 02:54:54,191 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-15 02:54:54,191 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:54,191 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:54,191 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:54,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:54,192 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-15 02:54:54,192 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-15 02:54:54,192 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34681,1681527209133' in zk 2023-04-15 02:54:54,192 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:54,192 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-15 02:54:54,193 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:54,193 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-15 02:54:54,193 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-15 02:54:54,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:54,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:54:54,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:54:54,193 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-15 02:54:54,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:54:54,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:54:54,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:54:54,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:54,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:54:54,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:54:54,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:54,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34681,1681527209133': 2023-04-15 02:54:54,195 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34681,1681527209133' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-15 02:54:54,195 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-15 02:54:54,195 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-15 02:54:54,195 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-15 02:54:54,195 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-15 02:54:54,195 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-15 02:54:54,196 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:54,196 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:54,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:54,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:54:54,196 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:54,196 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-15 02:54:54,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:54,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:54:54,196 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-15 02:54:54,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-15 02:54:54,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:54:54,197 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:54,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:54,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:54:54,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:54:54,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:54:54,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:54,198 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:54,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:54:54,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:54:54,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:54,199 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:54,199 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:54:54,199 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-15 02:54:54,200 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-15 02:54:54,199 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:54:54,200 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-15 02:54:54,200 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:54,200 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-15 02:54:54,200 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-15 02:54:54,200 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:54,200 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:54,200 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:54,200 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-15 02:54:54,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-15 02:54:54,200 DEBUG [Thread-1527] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-15 02:54:54,200 DEBUG [Thread-1527] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-15 02:54:54,200 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-15 02:54:54,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-15 02:54:54,300 DEBUG [Thread-1527] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-15 02:54:54,301 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-15 02:54:54,302 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-15 02:54:54,310 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-15 02:54:54,310 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-15 02:54:54,310 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-15 02:54:54,310 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-15 02:54:54,310 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-15 02:54:54,310 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-15 02:54:54,311 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:54,311 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:54:54,311 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-15 02:54:54,311 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-15 02:54:54,311 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:54,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-15 02:54:54,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:54:54,312 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:54,312 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-15 02:54:54,312 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:54,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-15 02:54:54,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:54:54,313 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-15 02:54:54,322 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-15 02:54:54,322 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-15 02:54:54,322 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-15 02:54:54,322 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-15 02:54:54,330 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-15 02:54:54,330 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-15 02:54:54,330 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681527274083.f8160290180ba126c904acc32b60eac7. 2023-04-15 02:54:54,330 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681527274083.f8160290180ba126c904acc32b60eac7. started... 2023-04-15 02:54:54,330 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681527274083.f8160290180ba126c904acc32b60eac7. as already flushing 2023-04-15 02:54:54,330 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681527274083.f8160290180ba126c904acc32b60eac7. 2023-04-15 02:54:54,330 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-15 02:54:54,330 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-15 02:54:54,330 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:54,330 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-15 02:54:54,331 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34681,1681527209133' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-15 02:54:54,332 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:54,332 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:54,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:54,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:54:54,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:54:54,332 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:54,332 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-15 02:54:54,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:54:54,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:54:54,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:54:54,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:54,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:54:54,333 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34681,1681527209133' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-15 02:54:54,334 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@fd9d077[Count = 0] remaining members to acquire global barrier 2023-04-15 02:54:54,334 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-15 02:54:54,334 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:54,334 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:54,334 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:54,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:54,335 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-15 02:54:54,335 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-15 02:54:54,335 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34681,1681527209133' in zk 2023-04-15 02:54:54,335 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-15 02:54:54,335 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-15 02:54:54,335 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-15 02:54:54,336 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:54,336 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34681,1681527209133' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-15 02:54:54,336 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-15 02:54:54,336 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-15 02:54:54,336 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-15 02:54:54,336 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-15 02:54:54,336 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-15 02:54:54,336 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-15 02:54:54,338 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:54,338 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:54,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:54,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:54:54,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:54:54,338 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:54,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:54,338 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-15 02:54:54,340 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-15 02:54:54,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-15 02:54:54,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:54,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:54:54,341 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:54,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:54:54,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:54:54,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:54:54,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:54,342 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:54,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:54:54,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:54:54,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:54,348 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-15 02:54:54,348 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-15 02:54:54,348 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:54,348 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-15 02:54:54,349 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-15 02:54:54,349 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-15 02:54:54,349 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-15 02:54:54,349 DEBUG [Thread-1527] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-15 02:54:54,349 DEBUG [Thread-1527] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-15 02:54:54,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-15 02:54:54,348 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:54:54,349 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:54:54,349 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:54,349 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:54,349 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:54,349 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:54,349 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-15 02:54:54,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-15 02:54:54,446 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f8160290180ba126c904acc32b60eac7, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:54:54,446 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34681] ipc.CallRunner(144): callId: 467 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:41918 deadline: 1681527354442, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f8160290180ba126c904acc32b60eac7, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:54,449 DEBUG [Thread-1527] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-15 02:54:54,449 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-15 02:54:54,451 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-15 02:54:54,451 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-15 02:54:54,451 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-15 02:54:54,451 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-15 02:54:54,451 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-15 02:54:54,452 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-15 02:54:54,452 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-15 02:54:54,452 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:54,452 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:54:54,453 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-15 02:54:54,453 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:54,453 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-15 02:54:54,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-15 02:54:54,453 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:54,454 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-15 02:54:54,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:54:54,454 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:54,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-15 02:54:54,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:54:54,454 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-15 02:54:54,454 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-15 02:54:54,454 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-15 02:54:54,454 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-15 02:54:54,454 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-15 02:54:54,454 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-15 02:54:54,455 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-15 02:54:54,455 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-15 02:54:54,455 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681527274083.f8160290180ba126c904acc32b60eac7. 2023-04-15 02:54:54,455 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681527274083.f8160290180ba126c904acc32b60eac7. started... 2023-04-15 02:54:54,455 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681527274083.f8160290180ba126c904acc32b60eac7. as already flushing 2023-04-15 02:54:54,455 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681527274083.f8160290180ba126c904acc32b60eac7. 2023-04-15 02:54:54,455 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-15 02:54:54,455 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-15 02:54:54,455 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:54,455 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-15 02:54:54,455 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34681,1681527209133' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-15 02:54:54,456 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:54,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:54,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:54:54,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:54:54,457 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:54,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:54:54,457 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:54,457 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-15 02:54:54,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:54:54,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:54:54,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:54,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:54:54,458 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34681,1681527209133' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-15 02:54:54,458 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3b315f18[Count = 0] remaining members to acquire global barrier 2023-04-15 02:54:54,458 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-15 02:54:54,458 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:54,458 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:54,459 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:54,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:54,459 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-15 02:54:54,459 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-15 02:54:54,459 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34681,1681527209133' in zk 2023-04-15 02:54:54,459 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:54,459 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-15 02:54:54,459 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:54,459 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-15 02:54:54,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:54,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:54:54,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:54:54,459 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-15 02:54:54,460 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-15 02:54:54,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:54:54,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:54:54,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:54:54,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:54,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:54:54,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:54:54,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:54,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34681,1681527209133': 2023-04-15 02:54:54,461 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34681,1681527209133' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-15 02:54:54,461 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-15 02:54:54,462 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-15 02:54:54,462 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-15 02:54:54,462 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-15 02:54:54,462 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-15 02:54:54,463 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:54,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:54,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:54:54,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:54:54,463 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:54,463 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-15 02:54:54,463 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:54,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:54,463 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-15 02:54:54,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-15 02:54:54,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:54:54,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:54,464 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:54,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:54:54,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:54:54,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:54:54,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:54,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:54:54,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:54:54,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:54,476 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:54,478 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:54,478 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:54:54,478 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:54:54,478 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:54,478 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:54,478 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:54,478 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:54,479 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-15 02:54:54,479 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-15 02:54:54,479 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-15 02:54:54,479 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-15 02:54:54,479 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-15 02:54:54,479 DEBUG [Thread-1527] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-15 02:54:54,479 DEBUG [Thread-1527] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-15 02:54:54,479 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-15 02:54:54,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-15 02:54:54,479 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-15 02:54:54,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-15 02:54:54,567 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=505 (bloomFilter=true), to=hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/f8160290180ba126c904acc32b60eac7/.tmp/C/039dae8ab93444db82666ec43736a590 2023-04-15 02:54:54,579 DEBUG [Thread-1527] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-15 02:54:54,580 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-15 02:54:54,581 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/f8160290180ba126c904acc32b60eac7/.tmp/A/8cc3365625a74bbe9ffca4472e1584e8 as hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/f8160290180ba126c904acc32b60eac7/A/8cc3365625a74bbe9ffca4472e1584e8 2023-04-15 02:54:54,581 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-15 02:54:54,587 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/f8160290180ba126c904acc32b60eac7/A/8cc3365625a74bbe9ffca4472e1584e8, entries=150, sequenceid=505, filesize=11.9 K 2023-04-15 02:54:54,588 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/f8160290180ba126c904acc32b60eac7/.tmp/B/805958efea2344fdb254357346ad44de as hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/f8160290180ba126c904acc32b60eac7/B/805958efea2344fdb254357346ad44de 2023-04-15 02:54:54,590 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-15 02:54:54,590 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-15 02:54:54,590 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-15 02:54:54,590 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-15 02:54:54,590 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-15 02:54:54,590 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-15 02:54:54,591 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:54,591 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:54:54,591 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-15 02:54:54,591 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:54,591 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-15 02:54:54,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-15 02:54:54,592 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:54,592 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-15 02:54:54,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:54:54,592 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:54,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-15 02:54:54,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:54:54,593 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-15 02:54:54,594 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-15 02:54:54,594 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-15 02:54:54,594 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-15 02:54:54,594 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-15 02:54:54,594 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-15 02:54:54,596 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/f8160290180ba126c904acc32b60eac7/B/805958efea2344fdb254357346ad44de, entries=100, sequenceid=505, filesize=9.5 K 2023-04-15 02:54:54,597 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/f8160290180ba126c904acc32b60eac7/.tmp/C/039dae8ab93444db82666ec43736a590 as hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/f8160290180ba126c904acc32b60eac7/C/039dae8ab93444db82666ec43736a590 2023-04-15 02:54:54,599 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-15 02:54:54,599 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681527274083.f8160290180ba126c904acc32b60eac7. 2023-04-15 02:54:54,599 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-15 02:54:54,599 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681527274083.f8160290180ba126c904acc32b60eac7. started... 2023-04-15 02:54:54,599 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681527274083.f8160290180ba126c904acc32b60eac7. as already flushing 2023-04-15 02:54:54,599 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681527274083.f8160290180ba126c904acc32b60eac7. 2023-04-15 02:54:54,600 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-15 02:54:54,600 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-15 02:54:54,600 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:54,600 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-15 02:54:54,600 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34681,1681527209133' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-15 02:54:54,602 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/f8160290180ba126c904acc32b60eac7/C/039dae8ab93444db82666ec43736a590, entries=100, sequenceid=505, filesize=9.5 K 2023-04-15 02:54:54,603 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=134.18 KB/137400 for f8160290180ba126c904acc32b60eac7 in 867ms, sequenceid=505, compaction requested=true 2023-04-15 02:54:54,603 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for f8160290180ba126c904acc32b60eac7: 2023-04-15 02:54:54,603 DEBUG [RS:0;jenkins-hbase20:34681-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-15 02:54:54,604 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-15 02:54:54,604 DEBUG [RS:0;jenkins-hbase20:34681-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-15 02:54:54,604 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-15 02:54:54,605 DEBUG [RS:0;jenkins-hbase20:34681-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37304 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-15 02:54:54,605 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-15 02:54:54,605 DEBUG [RS:0;jenkins-hbase20:34681-shortCompactions-0] regionserver.HStore(1912): f8160290180ba126c904acc32b60eac7/A is initiating minor compaction (all files) 2023-04-15 02:54:54,605 INFO [RS:0;jenkins-hbase20:34681-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of f8160290180ba126c904acc32b60eac7/A in TestAcidGuarantees,,1681527274083.f8160290180ba126c904acc32b60eac7. 2023-04-15 02:54:54,605 INFO [RS:0;jenkins-hbase20:34681-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/f8160290180ba126c904acc32b60eac7/A/9004437d3cc24f318e2e9d2d5d21a388, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/f8160290180ba126c904acc32b60eac7/A/465c74a754914f38b43b1f7bbea09d26, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/f8160290180ba126c904acc32b60eac7/A/8cc3365625a74bbe9ffca4472e1584e8] into tmpdir=hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/f8160290180ba126c904acc32b60eac7/.tmp, totalSize=36.4 K 2023-04-15 02:54:54,605 DEBUG [RS:0;jenkins-hbase20:34681-shortCompactions-0] compactions.Compactor(207): Compacting 9004437d3cc24f318e2e9d2d5d21a388, keycount=150, bloomtype=ROW, size=12.6 K, encoding=NONE, compression=NONE, seqNum=455, earliestPutTs=1681527289374 2023-04-15 02:54:54,606 DEBUG [RS:0;jenkins-hbase20:34681-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 34864 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-15 02:54:54,606 DEBUG [RS:0;jenkins-hbase20:34681-longCompactions-0] regionserver.HStore(1912): f8160290180ba126c904acc32b60eac7/B is initiating minor compaction (all files) 2023-04-15 02:54:54,606 INFO [RS:0;jenkins-hbase20:34681-longCompactions-0] regionserver.HRegion(2259): Starting compaction of f8160290180ba126c904acc32b60eac7/B in TestAcidGuarantees,,1681527274083.f8160290180ba126c904acc32b60eac7. 2023-04-15 02:54:54,606 INFO [RS:0;jenkins-hbase20:34681-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/f8160290180ba126c904acc32b60eac7/B/a9b63f57015d4c7ba5ccf3f0792317a5, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/f8160290180ba126c904acc32b60eac7/B/219e1fe89c37434394f36af8af7460ec, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/f8160290180ba126c904acc32b60eac7/B/805958efea2344fdb254357346ad44de] into tmpdir=hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/f8160290180ba126c904acc32b60eac7/.tmp, totalSize=34.0 K 2023-04-15 02:54:54,606 DEBUG [RS:0;jenkins-hbase20:34681-longCompactions-0] compactions.Compactor(207): Compacting a9b63f57015d4c7ba5ccf3f0792317a5, keycount=150, bloomtype=ROW, size=12.6 K, encoding=NONE, compression=NONE, seqNum=455, earliestPutTs=1681527289374 2023-04-15 02:54:54,606 DEBUG [RS:0;jenkins-hbase20:34681-shortCompactions-0] compactions.Compactor(207): Compacting 465c74a754914f38b43b1f7bbea09d26, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=492, earliestPutTs=1681527291583 2023-04-15 02:54:54,607 DEBUG [RS:0;jenkins-hbase20:34681-longCompactions-0] compactions.Compactor(207): Compacting 219e1fe89c37434394f36af8af7460ec, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=492, earliestPutTs=1681527291583 2023-04-15 02:54:54,607 DEBUG [RS:0;jenkins-hbase20:34681-longCompactions-0] compactions.Compactor(207): Compacting 805958efea2344fdb254357346ad44de, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=505, earliestPutTs=1681527293732 2023-04-15 02:54:54,607 DEBUG [RS:0;jenkins-hbase20:34681-shortCompactions-0] compactions.Compactor(207): Compacting 8cc3365625a74bbe9ffca4472e1584e8, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=505, earliestPutTs=1681527293732 2023-04-15 02:54:54,608 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:54,608 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:54,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:54,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:54:54,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:54:54,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:54:54,608 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:54,609 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-15 02:54:54,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:54:54,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:54:54,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:54,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:54:54,610 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34681,1681527209133' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-15 02:54:54,610 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-15 02:54:54,610 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@341cf266[Count = 0] remaining members to acquire global barrier 2023-04-15 02:54:54,610 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:54,613 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:54,614 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:54,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:54,614 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:54,614 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-15 02:54:54,614 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-15 02:54:54,614 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-15 02:54:54,614 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34681,1681527209133' in zk 2023-04-15 02:54:54,615 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-15 02:54:54,615 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-15 02:54:54,615 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:54,615 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-15 02:54:54,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:54,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:54:54,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:54:54,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:54:54,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:54:54,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:54:54,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:54,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:54:54,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:54:54,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:54,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34681,1681527209133': 2023-04-15 02:54:54,618 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34681,1681527209133' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-15 02:54:54,618 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-15 02:54:54,618 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-15 02:54:54,618 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-15 02:54:54,619 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-15 02:54:54,619 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-15 02:54:54,620 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:54,620 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:54,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:54,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:54:54,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:54:54,620 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:54,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:54,620 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-15 02:54:54,621 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-15 02:54:54,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-15 02:54:54,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:54,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:54:54,622 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:54,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:54:54,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:54:54,625 INFO [RS:0;jenkins-hbase20:34681-longCompactions-0] throttle.PressureAwareThroughputController(145): f8160290180ba126c904acc32b60eac7#B#compaction#254 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-15 02:54:54,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:54:54,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:54,626 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:54,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:54:54,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:54:54,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:54,632 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:54,632 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:54:54,632 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:54:54,632 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:54,632 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:54,632 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:54,632 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:54,632 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-15 02:54:54,632 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-15 02:54:54,632 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-15 02:54:54,632 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-15 02:54:54,633 DEBUG [Thread-1527] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-15 02:54:54,633 DEBUG [Thread-1527] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-15 02:54:54,634 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-15 02:54:54,634 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-15 02:54:54,634 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-15 02:54:54,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-15 02:54:54,635 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-15 02:54:54,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-15 02:54:54,653 INFO [RS:0;jenkins-hbase20:34681-shortCompactions-0] throttle.PressureAwareThroughputController(145): f8160290180ba126c904acc32b60eac7#A#compaction#255 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-15 02:54:54,675 DEBUG [RS:0;jenkins-hbase20:34681-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/f8160290180ba126c904acc32b60eac7/.tmp/B/f2c8ba28620940748c3055fb37f112be as hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/f8160290180ba126c904acc32b60eac7/B/f2c8ba28620940748c3055fb37f112be 2023-04-15 02:54:54,681 INFO [RS:0;jenkins-hbase20:34681-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in f8160290180ba126c904acc32b60eac7/B of f8160290180ba126c904acc32b60eac7 into f2c8ba28620940748c3055fb37f112be(size=12.7 K), total size for store is 12.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-15 02:54:54,681 DEBUG [RS:0;jenkins-hbase20:34681-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for f8160290180ba126c904acc32b60eac7: 2023-04-15 02:54:54,681 INFO [RS:0;jenkins-hbase20:34681-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681527274083.f8160290180ba126c904acc32b60eac7., storeName=f8160290180ba126c904acc32b60eac7/B, priority=13, startTime=1681527294604; duration=0sec 2023-04-15 02:54:54,681 DEBUG [RS:0;jenkins-hbase20:34681-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-15 02:54:54,681 DEBUG [RS:0;jenkins-hbase20:34681-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-15 02:54:54,683 DEBUG [RS:0;jenkins-hbase20:34681-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 34864 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-15 02:54:54,683 DEBUG [RS:0;jenkins-hbase20:34681-longCompactions-0] regionserver.HStore(1912): f8160290180ba126c904acc32b60eac7/C is initiating minor compaction (all files) 2023-04-15 02:54:54,684 INFO [RS:0;jenkins-hbase20:34681-longCompactions-0] regionserver.HRegion(2259): Starting compaction of f8160290180ba126c904acc32b60eac7/C in TestAcidGuarantees,,1681527274083.f8160290180ba126c904acc32b60eac7. 2023-04-15 02:54:54,684 INFO [RS:0;jenkins-hbase20:34681-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/f8160290180ba126c904acc32b60eac7/C/6f5b2fbcc01d4e2eb50f02d4ead85ce8, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/f8160290180ba126c904acc32b60eac7/C/ca8ca18f22304064843cb86197b6fb9f, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/f8160290180ba126c904acc32b60eac7/C/039dae8ab93444db82666ec43736a590] into tmpdir=hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/f8160290180ba126c904acc32b60eac7/.tmp, totalSize=34.0 K 2023-04-15 02:54:54,684 DEBUG [RS:0;jenkins-hbase20:34681-longCompactions-0] compactions.Compactor(207): Compacting 6f5b2fbcc01d4e2eb50f02d4ead85ce8, keycount=150, bloomtype=ROW, size=12.6 K, encoding=NONE, compression=NONE, seqNum=455, earliestPutTs=1681527289374 2023-04-15 02:54:54,684 DEBUG [RS:0;jenkins-hbase20:34681-longCompactions-0] compactions.Compactor(207): Compacting ca8ca18f22304064843cb86197b6fb9f, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=492, earliestPutTs=1681527291583 2023-04-15 02:54:54,685 DEBUG [RS:0;jenkins-hbase20:34681-longCompactions-0] compactions.Compactor(207): Compacting 039dae8ab93444db82666ec43736a590, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=505, earliestPutTs=1681527293732 2023-04-15 02:54:54,691 INFO [RS:0;jenkins-hbase20:34681-longCompactions-0] throttle.PressureAwareThroughputController(145): f8160290180ba126c904acc32b60eac7#C#compaction#256 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-15 02:54:54,733 DEBUG [Thread-1527] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-15 02:54:54,733 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-15 02:54:54,735 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-15 02:54:54,736 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-15 02:54:54,736 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-15 02:54:54,736 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-15 02:54:54,736 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-15 02:54:54,736 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-15 02:54:54,736 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-15 02:54:54,736 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:54,736 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:54:54,737 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:54,737 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-15 02:54:54,737 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-15 02:54:54,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-15 02:54:54,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:54:54,737 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:54,737 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-15 02:54:54,737 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:54,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-15 02:54:54,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:54:54,738 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-15 02:54:54,738 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-15 02:54:54,738 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-15 02:54:54,738 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-15 02:54:54,738 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-15 02:54:54,738 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-15 02:54:54,738 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-15 02:54:54,738 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-15 02:54:54,738 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681527274083.f8160290180ba126c904acc32b60eac7. 2023-04-15 02:54:54,738 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681527274083.f8160290180ba126c904acc32b60eac7. started... 2023-04-15 02:54:54,739 INFO [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing f8160290180ba126c904acc32b60eac7 3/3 column families, dataSize=134.18 KB heapSize=384.98 KB 2023-04-15 02:54:54,739 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK f8160290180ba126c904acc32b60eac7, store=A 2023-04-15 02:54:54,739 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-15 02:54:54,739 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK f8160290180ba126c904acc32b60eac7, store=B 2023-04-15 02:54:54,739 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-15 02:54:54,739 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK f8160290180ba126c904acc32b60eac7, store=C 2023-04-15 02:54:54,739 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-15 02:54:54,951 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34681] regionserver.HRegion(9158): Flush requested on f8160290180ba126c904acc32b60eac7 2023-04-15 02:54:54,951 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681527274083.f8160290180ba126c904acc32b60eac7. as already flushing 2023-04-15 02:54:54,969 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f8160290180ba126c904acc32b60eac7, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:54:54,969 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34681] ipc.CallRunner(144): callId: 476 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:41918 deadline: 1681527354967, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f8160290180ba126c904acc32b60eac7, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:55,072 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f8160290180ba126c904acc32b60eac7, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:54:55,072 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34681] ipc.CallRunner(144): callId: 478 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:41918 deadline: 1681527355070, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f8160290180ba126c904acc32b60eac7, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:55,095 DEBUG [RS:0;jenkins-hbase20:34681-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/f8160290180ba126c904acc32b60eac7/.tmp/A/b86d1b177a2d4ecfb61857597a644ad0 as hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/f8160290180ba126c904acc32b60eac7/A/b86d1b177a2d4ecfb61857597a644ad0 2023-04-15 02:54:55,100 INFO [RS:0;jenkins-hbase20:34681-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in f8160290180ba126c904acc32b60eac7/A of f8160290180ba126c904acc32b60eac7 into b86d1b177a2d4ecfb61857597a644ad0(size=12.7 K), total size for store is 12.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-15 02:54:55,101 DEBUG [RS:0;jenkins-hbase20:34681-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for f8160290180ba126c904acc32b60eac7: 2023-04-15 02:54:55,101 INFO [RS:0;jenkins-hbase20:34681-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681527274083.f8160290180ba126c904acc32b60eac7., storeName=f8160290180ba126c904acc32b60eac7/A, priority=13, startTime=1681527294603; duration=0sec 2023-04-15 02:54:55,101 DEBUG [RS:0;jenkins-hbase20:34681-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-15 02:54:55,116 DEBUG [RS:0;jenkins-hbase20:34681-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/f8160290180ba126c904acc32b60eac7/.tmp/C/49468c20795140b7bd23c7e29c1bd3e2 as hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/f8160290180ba126c904acc32b60eac7/C/49468c20795140b7bd23c7e29c1bd3e2 2023-04-15 02:54:55,121 INFO [RS:0;jenkins-hbase20:34681-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in f8160290180ba126c904acc32b60eac7/C of f8160290180ba126c904acc32b60eac7 into 49468c20795140b7bd23c7e29c1bd3e2(size=12.7 K), total size for store is 12.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-15 02:54:55,121 DEBUG [RS:0;jenkins-hbase20:34681-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for f8160290180ba126c904acc32b60eac7: 2023-04-15 02:54:55,121 INFO [RS:0;jenkins-hbase20:34681-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681527274083.f8160290180ba126c904acc32b60eac7., storeName=f8160290180ba126c904acc32b60eac7/C, priority=13, startTime=1681527294605; duration=0sec 2023-04-15 02:54:55,121 DEBUG [RS:0;jenkins-hbase20:34681-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-15 02:54:55,164 INFO [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=543 (bloomFilter=true), to=hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/f8160290180ba126c904acc32b60eac7/.tmp/A/93b4a56e01ad4055853465b9cbe01d96 2023-04-15 02:54:55,198 INFO [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=543 (bloomFilter=true), to=hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/f8160290180ba126c904acc32b60eac7/.tmp/B/ef70ec718f9246668a85fbea43b0985b 2023-04-15 02:54:55,242 INFO [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=543 (bloomFilter=true), to=hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/f8160290180ba126c904acc32b60eac7/.tmp/C/f64cc99821ad4f14926e330d9a735d6b 2023-04-15 02:54:55,248 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/f8160290180ba126c904acc32b60eac7/.tmp/A/93b4a56e01ad4055853465b9cbe01d96 as hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/f8160290180ba126c904acc32b60eac7/A/93b4a56e01ad4055853465b9cbe01d96 2023-04-15 02:54:55,255 INFO [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/f8160290180ba126c904acc32b60eac7/A/93b4a56e01ad4055853465b9cbe01d96, entries=150, sequenceid=543, filesize=11.9 K 2023-04-15 02:54:55,256 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/f8160290180ba126c904acc32b60eac7/.tmp/B/ef70ec718f9246668a85fbea43b0985b as hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/f8160290180ba126c904acc32b60eac7/B/ef70ec718f9246668a85fbea43b0985b 2023-04-15 02:54:55,275 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f8160290180ba126c904acc32b60eac7, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:54:55,275 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34681] ipc.CallRunner(144): callId: 480 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:41918 deadline: 1681527355273, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f8160290180ba126c904acc32b60eac7, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:55,276 INFO [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/f8160290180ba126c904acc32b60eac7/B/ef70ec718f9246668a85fbea43b0985b, entries=150, sequenceid=543, filesize=11.9 K 2023-04-15 02:54:55,277 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/f8160290180ba126c904acc32b60eac7/.tmp/C/f64cc99821ad4f14926e330d9a735d6b as hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/f8160290180ba126c904acc32b60eac7/C/f64cc99821ad4f14926e330d9a735d6b 2023-04-15 02:54:55,282 INFO [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/f8160290180ba126c904acc32b60eac7/C/f64cc99821ad4f14926e330d9a735d6b, entries=150, sequenceid=543, filesize=11.9 K 2023-04-15 02:54:55,283 INFO [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~134.18 KB/137400, heapSize ~384.94 KB/394176, currentSize=46.96 KB/48090 for f8160290180ba126c904acc32b60eac7 in 544ms, sequenceid=543, compaction requested=false 2023-04-15 02:54:55,283 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for f8160290180ba126c904acc32b60eac7: 2023-04-15 02:54:55,283 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681527274083.f8160290180ba126c904acc32b60eac7. 2023-04-15 02:54:55,283 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-15 02:54:55,283 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-15 02:54:55,283 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:55,286 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-15 02:54:55,286 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34681,1681527209133' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-15 02:54:55,287 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:55,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:55,287 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:55,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:54:55,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:54:55,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:54:55,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:54:55,288 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:55,288 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-15 02:54:55,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:54:55,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:55,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:54:55,289 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34681,1681527209133' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-15 02:54:55,289 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4c1915e4[Count = 0] remaining members to acquire global barrier 2023-04-15 02:54:55,289 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-15 02:54:55,289 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:55,290 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:55,291 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:55,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:55,291 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:55,291 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-15 02:54:55,291 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-15 02:54:55,291 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-15 02:54:55,291 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34681,1681527209133' in zk 2023-04-15 02:54:55,292 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-15 02:54:55,292 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-15 02:54:55,292 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-15 02:54:55,292 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:55,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:55,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:54:55,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:54:55,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:54:55,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:54:55,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:54:55,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:55,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:54:55,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:54:55,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:55,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34681,1681527209133': 2023-04-15 02:54:55,294 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34681,1681527209133' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-15 02:54:55,294 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-15 02:54:55,294 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-15 02:54:55,294 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-15 02:54:55,294 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-15 02:54:55,294 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-15 02:54:55,295 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:55,295 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-15 02:54:55,295 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:55,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:55,296 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:55,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:55,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:54:55,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:54:55,296 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-15 02:54:55,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-15 02:54:55,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:55,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:54:55,297 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:55,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:54:55,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:54:55,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:54:55,299 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:55,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:55,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:54:55,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:54:55,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:55,307 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-15 02:54:55,307 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:55,307 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-15 02:54:55,307 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-15 02:54:55,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-15 02:54:55,307 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:54:55,307 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-15 02:54:55,307 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-15 02:54:55,307 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-15 02:54:55,307 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-15 02:54:55,307 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-15 02:54:55,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-15 02:54:55,307 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:54:55,307 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:55,307 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:55,307 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:55,307 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:55,307 DEBUG [Thread-1527] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-15 02:54:55,307 DEBUG [Thread-1527] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-15 02:54:55,408 DEBUG [Thread-1527] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-15 02:54:55,408 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-15 02:54:55,410 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-15 02:54:55,411 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-15 02:54:55,411 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-15 02:54:55,411 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-15 02:54:55,411 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-15 02:54:55,411 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-15 02:54:55,411 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-15 02:54:55,411 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:55,411 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:54:55,412 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:55,412 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-15 02:54:55,412 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-15 02:54:55,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-15 02:54:55,412 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:55,412 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-15 02:54:55,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:54:55,413 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:55,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-15 02:54:55,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:54:55,413 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-15 02:54:55,413 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-15 02:54:55,413 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-15 02:54:55,413 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-15 02:54:55,413 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-15 02:54:55,413 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-15 02:54:55,414 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-15 02:54:55,414 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-15 02:54:55,414 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681527274083.f8160290180ba126c904acc32b60eac7. 2023-04-15 02:54:55,414 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681527274083.f8160290180ba126c904acc32b60eac7. started... 2023-04-15 02:54:55,414 INFO [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing f8160290180ba126c904acc32b60eac7 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-15 02:54:55,414 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK f8160290180ba126c904acc32b60eac7, store=A 2023-04-15 02:54:55,414 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-15 02:54:55,414 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK f8160290180ba126c904acc32b60eac7, store=B 2023-04-15 02:54:55,414 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-15 02:54:55,414 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK f8160290180ba126c904acc32b60eac7, store=C 2023-04-15 02:54:55,414 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-15 02:54:55,432 INFO [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=555 (bloomFilter=true), to=hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/f8160290180ba126c904acc32b60eac7/.tmp/A/23a2c20460fa4566a862fab3caa628ad 2023-04-15 02:54:55,454 INFO [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=555 (bloomFilter=true), to=hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/f8160290180ba126c904acc32b60eac7/.tmp/B/4b316212595c42cdb3390715265e9446 2023-04-15 02:54:55,469 INFO [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=555 (bloomFilter=true), to=hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/f8160290180ba126c904acc32b60eac7/.tmp/C/619b1f4e5ede48f18c9511b08d2d11f8 2023-04-15 02:54:55,474 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/f8160290180ba126c904acc32b60eac7/.tmp/A/23a2c20460fa4566a862fab3caa628ad as hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/f8160290180ba126c904acc32b60eac7/A/23a2c20460fa4566a862fab3caa628ad 2023-04-15 02:54:55,478 INFO [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/f8160290180ba126c904acc32b60eac7/A/23a2c20460fa4566a862fab3caa628ad, entries=150, sequenceid=555, filesize=11.9 K 2023-04-15 02:54:55,479 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/f8160290180ba126c904acc32b60eac7/.tmp/B/4b316212595c42cdb3390715265e9446 as hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/f8160290180ba126c904acc32b60eac7/B/4b316212595c42cdb3390715265e9446 2023-04-15 02:54:55,490 INFO [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/f8160290180ba126c904acc32b60eac7/B/4b316212595c42cdb3390715265e9446, entries=150, sequenceid=555, filesize=11.9 K 2023-04-15 02:54:55,493 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/f8160290180ba126c904acc32b60eac7/.tmp/C/619b1f4e5ede48f18c9511b08d2d11f8 as hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/f8160290180ba126c904acc32b60eac7/C/619b1f4e5ede48f18c9511b08d2d11f8 2023-04-15 02:54:55,497 INFO [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/f8160290180ba126c904acc32b60eac7/C/619b1f4e5ede48f18c9511b08d2d11f8, entries=150, sequenceid=555, filesize=11.9 K 2023-04-15 02:54:55,498 INFO [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=0 B/0 for f8160290180ba126c904acc32b60eac7 in 84ms, sequenceid=555, compaction requested=true 2023-04-15 02:54:55,498 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for f8160290180ba126c904acc32b60eac7: 2023-04-15 02:54:55,498 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681527274083.f8160290180ba126c904acc32b60eac7. 2023-04-15 02:54:55,498 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-15 02:54:55,498 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-15 02:54:55,498 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:55,498 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-15 02:54:55,498 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34681,1681527209133' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-15 02:54:55,499 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:55,499 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:55,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:55,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:54:55,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:54:55,499 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:55,499 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-15 02:54:55,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:54:55,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:54:55,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:54:55,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:55,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:54:55,501 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34681,1681527209133' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-15 02:54:55,501 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@d822e78[Count = 0] remaining members to acquire global barrier 2023-04-15 02:54:55,501 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-15 02:54:55,501 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:55,502 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:55,502 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:55,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:55,502 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-15 02:54:55,502 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-15 02:54:55,502 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34681,1681527209133' in zk 2023-04-15 02:54:55,502 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-15 02:54:55,503 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-15 02:54:55,503 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:55,503 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-15 02:54:55,503 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34681,1681527209133' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-15 02:54:55,503 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-15 02:54:55,503 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-15 02:54:55,503 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-15 02:54:55,503 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-15 02:54:55,503 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-15 02:54:55,503 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-15 02:54:55,504 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:55,504 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:55,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:55,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:54:55,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:54:55,504 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:55,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:55,504 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-15 02:54:55,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:54:55,504 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:55,505 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-15 02:54:55,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-15 02:54:55,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:54:55,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:55,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:54:55,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:54:55,506 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:55,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:55,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:54:55,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:54:55,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:55,507 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-15 02:54:55,507 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-15 02:54:55,507 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:55,507 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:54:55,507 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-15 02:54:55,508 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-15 02:54:55,507 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-15 02:54:55,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-15 02:54:55,507 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-15 02:54:55,507 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:54:55,508 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:55,508 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:55,508 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:55,508 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:55,507 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-15 02:54:55,508 DEBUG [Thread-1527] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-15 02:54:55,508 DEBUG [Thread-1527] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-15 02:54:55,508 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-15 02:54:55,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-15 02:54:55,589 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34681] regionserver.HRegion(9158): Flush requested on f8160290180ba126c904acc32b60eac7 2023-04-15 02:54:55,589 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing f8160290180ba126c904acc32b60eac7 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-15 02:54:55,590 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK f8160290180ba126c904acc32b60eac7, store=A 2023-04-15 02:54:55,590 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-15 02:54:55,590 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK f8160290180ba126c904acc32b60eac7, store=B 2023-04-15 02:54:55,590 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-15 02:54:55,590 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK f8160290180ba126c904acc32b60eac7, store=C 2023-04-15 02:54:55,590 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-15 02:54:55,608 DEBUG [Thread-1527] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-15 02:54:55,608 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-15 02:54:55,610 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-15 02:54:55,611 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-15 02:54:55,611 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-15 02:54:55,611 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-15 02:54:55,611 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-15 02:54:55,611 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-15 02:54:55,611 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-15 02:54:55,611 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:55,611 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:54:55,612 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-15 02:54:55,612 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:55,612 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-15 02:54:55,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-15 02:54:55,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:54:55,612 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:55,612 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-15 02:54:55,612 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:55,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-15 02:54:55,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:54:55,613 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-15 02:54:55,613 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-15 02:54:55,613 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-15 02:54:55,613 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-15 02:54:55,613 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-15 02:54:55,613 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-15 02:54:55,614 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-15 02:54:55,614 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-15 02:54:55,614 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681527274083.f8160290180ba126c904acc32b60eac7. 2023-04-15 02:54:55,614 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681527274083.f8160290180ba126c904acc32b60eac7. started... 2023-04-15 02:54:55,614 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681527274083.f8160290180ba126c904acc32b60eac7. as already flushing 2023-04-15 02:54:55,614 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681527274083.f8160290180ba126c904acc32b60eac7. 2023-04-15 02:54:55,614 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-15 02:54:55,614 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-15 02:54:55,614 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:55,614 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-15 02:54:55,614 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34681,1681527209133' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-15 02:54:55,615 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:55,615 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:55,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:55,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:54:55,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:54:55,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:54:55,615 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:55,615 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-15 02:54:55,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:54:55,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:54:55,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:55,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:54:55,616 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34681,1681527209133' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-15 02:54:55,616 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3c7a694f[Count = 0] remaining members to acquire global barrier 2023-04-15 02:54:55,616 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-15 02:54:55,616 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:55,617 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:55,617 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:55,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:55,617 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-15 02:54:55,617 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:55,617 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-15 02:54:55,617 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-15 02:54:55,617 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34681,1681527209133' in zk 2023-04-15 02:54:55,618 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-15 02:54:55,618 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-15 02:54:55,618 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:55,618 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-15 02:54:55,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:55,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:54:55,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:54:55,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:54:55,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:54:55,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:54:55,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:55,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:54:55,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:54:55,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:55,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34681,1681527209133': 2023-04-15 02:54:55,620 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34681,1681527209133' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-15 02:54:55,620 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-15 02:54:55,620 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-15 02:54:55,620 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-15 02:54:55,620 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-15 02:54:55,620 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-15 02:54:55,621 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:55,621 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:55,621 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-15 02:54:55,621 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:55,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:55,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:55,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:54:55,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:54:55,621 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-15 02:54:55,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-15 02:54:55,621 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:55,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:54:55,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:55,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:54:55,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:54:55,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:54:55,622 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:55,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:55,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:54:55,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:54:55,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:55,624 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:55,625 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:54:55,625 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:54:55,625 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:55,625 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:55,625 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:55,625 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:55,625 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-15 02:54:55,625 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-15 02:54:55,625 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-15 02:54:55,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-15 02:54:55,625 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-15 02:54:55,625 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-15 02:54:55,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-15 02:54:55,625 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-15 02:54:55,625 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-15 02:54:55,625 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-15 02:54:55,625 DEBUG [Thread-1527] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-15 02:54:55,626 DEBUG [Thread-1527] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-15 02:54:55,633 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=565 (bloomFilter=true), to=hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/f8160290180ba126c904acc32b60eac7/.tmp/A/c646a2bd66f74fe08577384895aa253b 2023-04-15 02:54:55,661 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681527295643, max timestamp=1681527295658Num uniques 859; 2023-04-15 02:54:55,661 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-15 02:54:55,662 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681527295643, max timestamp=1681527295658Num uniques 859; 2023-04-15 02:54:55,662 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-2] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-15 02:54:55,662 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681527295643, max timestamp=1681527295658Num uniques 859; 2023-04-15 02:54:55,662 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-9] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-15 02:54:55,725 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f8160290180ba126c904acc32b60eac7, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:54:55,725 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34681] ipc.CallRunner(144): callId: 523 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:41918 deadline: 1681527355722, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f8160290180ba126c904acc32b60eac7, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:55,726 DEBUG [Thread-1527] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-15 02:54:55,726 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-15 02:54:55,727 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-15 02:54:55,728 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-15 02:54:55,728 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-15 02:54:55,728 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-15 02:54:55,728 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-15 02:54:55,729 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-15 02:54:55,729 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-15 02:54:55,729 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:55,729 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:54:55,730 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:55,730 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-15 02:54:55,730 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-15 02:54:55,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-15 02:54:55,730 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:55,730 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-15 02:54:55,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:54:55,730 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:55,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-15 02:54:55,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:54:55,731 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-15 02:54:55,734 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-15 02:54:55,734 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-15 02:54:55,734 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-15 02:54:55,734 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-15 02:54:55,734 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-15 02:54:55,734 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-15 02:54:55,734 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-15 02:54:55,734 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681527274083.f8160290180ba126c904acc32b60eac7. 2023-04-15 02:54:55,734 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681527274083.f8160290180ba126c904acc32b60eac7. started... 2023-04-15 02:54:55,734 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681527274083.f8160290180ba126c904acc32b60eac7. as already flushing 2023-04-15 02:54:55,734 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681527274083.f8160290180ba126c904acc32b60eac7. 2023-04-15 02:54:55,735 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-15 02:54:55,735 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-15 02:54:55,735 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:55,735 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-15 02:54:55,735 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34681,1681527209133' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-15 02:54:55,736 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:55,736 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:55,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:55,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:54:55,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:54:55,736 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:55,736 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-15 02:54:55,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:54:55,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:54:55,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:54:55,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:55,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:54:55,738 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34681,1681527209133' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-15 02:54:55,738 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-15 02:54:55,738 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6fa63364[Count = 0] remaining members to acquire global barrier 2023-04-15 02:54:55,738 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:55,738 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:55,738 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:55,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:55,739 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-15 02:54:55,739 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:55,739 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-15 02:54:55,739 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-15 02:54:55,739 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34681,1681527209133' in zk 2023-04-15 02:54:55,739 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:55,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:55,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:54:55,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:54:55,740 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-15 02:54:55,740 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-15 02:54:55,740 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-15 02:54:55,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:54:55,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:54:55,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:54:55,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:55,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:54:55,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:54:55,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:55,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34681,1681527209133': 2023-04-15 02:54:55,742 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34681,1681527209133' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-15 02:54:55,742 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-15 02:54:55,742 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-15 02:54:55,742 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-15 02:54:55,742 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-15 02:54:55,742 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-15 02:54:55,743 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:55,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:55,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:54:55,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:54:55,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:54:55,744 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:55,744 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-15 02:54:55,744 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:55,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:55,744 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:55,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:54:55,744 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-15 02:54:55,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-15 02:54:55,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:54:55,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:55,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:54:55,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:55,745 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:55,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:54:55,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:54:55,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:55,747 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-15 02:54:55,747 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-15 02:54:55,747 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-15 02:54:55,747 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-15 02:54:55,747 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-15 02:54:55,747 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-15 02:54:55,747 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-15 02:54:55,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-15 02:54:55,747 DEBUG [Thread-1527] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-15 02:54:55,747 DEBUG [Thread-1527] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-15 02:54:55,747 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:55,747 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:54:55,747 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:54:55,748 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:55,748 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:55,748 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:55,748 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:55,748 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-15 02:54:55,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-15 02:54:55,827 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f8160290180ba126c904acc32b60eac7, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:54:55,827 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34681] ipc.CallRunner(144): callId: 525 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:41918 deadline: 1681527355826, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f8160290180ba126c904acc32b60eac7, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:55,847 DEBUG [Thread-1527] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-15 02:54:55,848 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-15 02:54:55,850 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-15 02:54:55,854 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-15 02:54:55,855 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-15 02:54:55,855 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-15 02:54:55,855 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-15 02:54:55,855 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-15 02:54:55,855 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-15 02:54:55,855 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:55,855 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:54:55,856 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:55,856 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-15 02:54:55,856 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-15 02:54:55,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-15 02:54:55,856 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:55,856 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-15 02:54:55,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:54:55,857 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:55,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-15 02:54:55,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:54:55,857 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-15 02:54:55,857 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-15 02:54:55,857 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-15 02:54:55,857 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-15 02:54:55,857 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-15 02:54:55,857 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-15 02:54:55,858 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-15 02:54:55,858 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-15 02:54:55,858 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681527274083.f8160290180ba126c904acc32b60eac7. 2023-04-15 02:54:55,858 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681527274083.f8160290180ba126c904acc32b60eac7. started... 2023-04-15 02:54:55,858 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681527274083.f8160290180ba126c904acc32b60eac7. as already flushing 2023-04-15 02:54:55,858 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681527274083.f8160290180ba126c904acc32b60eac7. 2023-04-15 02:54:55,858 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-15 02:54:55,858 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-15 02:54:55,858 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:55,858 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-15 02:54:55,858 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34681,1681527209133' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-15 02:54:55,859 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:55,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:55,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:54:55,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:54:55,859 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:55,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:54:55,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:54:55,860 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:55,860 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-15 02:54:55,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:54:55,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:55,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:54:55,860 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34681,1681527209133' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-15 02:54:55,860 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6b94ad41[Count = 0] remaining members to acquire global barrier 2023-04-15 02:54:55,860 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-15 02:54:55,860 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:55,861 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:55,861 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:55,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:55,861 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-15 02:54:55,861 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-15 02:54:55,861 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34681,1681527209133' in zk 2023-04-15 02:54:55,861 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:55,861 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-15 02:54:55,862 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:55,862 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-15 02:54:55,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:55,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:54:55,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:54:55,862 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-15 02:54:55,862 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-15 02:54:55,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:54:55,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:54:55,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:54:55,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:55,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:54:55,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:54:55,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:55,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34681,1681527209133': 2023-04-15 02:54:55,864 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34681,1681527209133' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-15 02:54:55,864 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-15 02:54:55,864 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-15 02:54:55,864 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-15 02:54:55,864 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-15 02:54:55,864 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-15 02:54:55,865 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:55,865 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-15 02:54:55,865 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:55,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:55,865 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-15 02:54:55,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-15 02:54:55,865 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:55,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:55,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:54:55,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:54:55,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:55,866 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:55,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:54:55,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:54:55,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:54:55,867 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:55,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:54:55,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:55,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:54:55,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:54:55,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:55,881 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-15 02:54:55,882 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-15 02:54:55,882 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-15 02:54:55,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-15 02:54:55,882 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-15 02:54:55,882 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-15 02:54:55,882 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-15 02:54:55,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-15 02:54:55,882 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-15 02:54:55,882 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:55,882 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:54:55,882 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:54:55,882 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:55,882 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:55,882 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:55,882 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:55,882 DEBUG [Thread-1527] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-15 02:54:55,882 DEBUG [Thread-1527] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-15 02:54:55,883 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-15 02:54:55,983 DEBUG [Thread-1527] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-15 02:54:55,983 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-15 02:54:55,984 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-15 02:54:55,985 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-15 02:54:55,985 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-15 02:54:55,986 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-15 02:54:55,986 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-15 02:54:55,986 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-15 02:54:55,986 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-15 02:54:55,986 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:55,986 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:54:55,987 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:55,987 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-15 02:54:55,987 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-15 02:54:55,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-15 02:54:55,987 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:55,987 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-15 02:54:55,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:54:55,987 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:55,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-15 02:54:55,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:54:55,988 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-15 02:54:55,988 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-15 02:54:55,989 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-15 02:54:55,989 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-15 02:54:55,989 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-15 02:54:55,989 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-15 02:54:55,989 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-15 02:54:55,989 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-15 02:54:55,989 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681527274083.f8160290180ba126c904acc32b60eac7. 2023-04-15 02:54:55,989 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681527274083.f8160290180ba126c904acc32b60eac7. started... 2023-04-15 02:54:55,989 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681527274083.f8160290180ba126c904acc32b60eac7. as already flushing 2023-04-15 02:54:55,989 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681527274083.f8160290180ba126c904acc32b60eac7. 2023-04-15 02:54:55,989 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-15 02:54:55,989 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-15 02:54:55,989 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:55,989 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-15 02:54:55,989 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34681,1681527209133' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-15 02:54:55,990 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:55,990 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:55,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:55,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:54:55,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:54:55,990 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:55,991 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-15 02:54:55,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:54:55,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:54:55,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:54:55,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:55,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:54:55,992 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34681,1681527209133' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-15 02:54:55,992 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@77f9bfa5[Count = 0] remaining members to acquire global barrier 2023-04-15 02:54:55,992 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-15 02:54:55,992 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:55,993 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:55,993 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:55,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:55,993 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:55,993 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-15 02:54:55,993 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-15 02:54:55,993 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-15 02:54:55,993 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34681,1681527209133' in zk 2023-04-15 02:54:55,994 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:55,994 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-15 02:54:55,994 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-15 02:54:55,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:55,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:54:55,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:54:55,994 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-15 02:54:55,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:54:55,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:54:55,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:54:55,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:55,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:54:55,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:54:55,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:55,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34681,1681527209133': 2023-04-15 02:54:55,999 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34681,1681527209133' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-15 02:54:55,999 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-15 02:54:55,999 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-15 02:54:55,999 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-15 02:54:55,999 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-15 02:54:55,999 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-15 02:54:56,006 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:56,007 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:56,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:56,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:54:56,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:54:56,007 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-15 02:54:56,007 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:56,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:56,007 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:56,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:54:56,007 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-15 02:54:56,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-15 02:54:56,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:54:56,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:56,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:54:56,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:54:56,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:56,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:54:56,010 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:56,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:54:56,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:56,015 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-15 02:54:56,015 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:56,015 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:54:56,016 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:54:56,016 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:56,016 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:56,016 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:56,016 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-15 02:54:56,015 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-15 02:54:56,016 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-15 02:54:56,016 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-15 02:54:56,016 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-15 02:54:56,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-15 02:54:56,016 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:56,016 DEBUG [Thread-1527] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-15 02:54:56,016 DEBUG [Thread-1527] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-15 02:54:56,016 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-15 02:54:56,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-15 02:54:56,028 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f8160290180ba126c904acc32b60eac7, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:54:56,029 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34681] ipc.CallRunner(144): callId: 527 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:41918 deadline: 1681527356028, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f8160290180ba126c904acc32b60eac7, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:56,046 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=565 (bloomFilter=true), to=hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/f8160290180ba126c904acc32b60eac7/.tmp/B/6c158891089b4e7491d4f81163358b6e 2023-04-15 02:54:56,067 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=565 (bloomFilter=true), to=hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/f8160290180ba126c904acc32b60eac7/.tmp/C/ecc6d46705bd4be588a7418471c777ec 2023-04-15 02:54:56,072 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/f8160290180ba126c904acc32b60eac7/.tmp/A/c646a2bd66f74fe08577384895aa253b as hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/f8160290180ba126c904acc32b60eac7/A/c646a2bd66f74fe08577384895aa253b 2023-04-15 02:54:56,076 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/f8160290180ba126c904acc32b60eac7/A/c646a2bd66f74fe08577384895aa253b, entries=150, sequenceid=565, filesize=11.9 K 2023-04-15 02:54:56,077 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/f8160290180ba126c904acc32b60eac7/.tmp/B/6c158891089b4e7491d4f81163358b6e as hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/f8160290180ba126c904acc32b60eac7/B/6c158891089b4e7491d4f81163358b6e 2023-04-15 02:54:56,083 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/f8160290180ba126c904acc32b60eac7/B/6c158891089b4e7491d4f81163358b6e, entries=150, sequenceid=565, filesize=11.9 K 2023-04-15 02:54:56,083 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/f8160290180ba126c904acc32b60eac7/.tmp/C/ecc6d46705bd4be588a7418471c777ec as hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/f8160290180ba126c904acc32b60eac7/C/ecc6d46705bd4be588a7418471c777ec 2023-04-15 02:54:56,095 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/f8160290180ba126c904acc32b60eac7/C/ecc6d46705bd4be588a7418471c777ec, entries=150, sequenceid=565, filesize=11.9 K 2023-04-15 02:54:56,096 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=134.18 KB/137400 for f8160290180ba126c904acc32b60eac7 in 507ms, sequenceid=565, compaction requested=true 2023-04-15 02:54:56,096 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for f8160290180ba126c904acc32b60eac7: 2023-04-15 02:54:56,096 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-15 02:54:56,096 DEBUG [RS:0;jenkins-hbase20:34681-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-15 02:54:56,097 DEBUG [RS:0;jenkins-hbase20:34681-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-15 02:54:56,096 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-15 02:54:56,097 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-15 02:54:56,099 DEBUG [RS:0;jenkins-hbase20:34681-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 49596 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-15 02:54:56,099 DEBUG [RS:0;jenkins-hbase20:34681-shortCompactions-0] regionserver.HStore(1912): f8160290180ba126c904acc32b60eac7/A is initiating minor compaction (all files) 2023-04-15 02:54:56,099 INFO [RS:0;jenkins-hbase20:34681-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of f8160290180ba126c904acc32b60eac7/A in TestAcidGuarantees,,1681527274083.f8160290180ba126c904acc32b60eac7. 2023-04-15 02:54:56,099 INFO [RS:0;jenkins-hbase20:34681-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/f8160290180ba126c904acc32b60eac7/A/b86d1b177a2d4ecfb61857597a644ad0, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/f8160290180ba126c904acc32b60eac7/A/93b4a56e01ad4055853465b9cbe01d96, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/f8160290180ba126c904acc32b60eac7/A/23a2c20460fa4566a862fab3caa628ad, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/f8160290180ba126c904acc32b60eac7/A/c646a2bd66f74fe08577384895aa253b] into tmpdir=hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/f8160290180ba126c904acc32b60eac7/.tmp, totalSize=48.4 K 2023-04-15 02:54:56,100 DEBUG [RS:0;jenkins-hbase20:34681-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 49596 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-15 02:54:56,100 DEBUG [RS:0;jenkins-hbase20:34681-longCompactions-0] regionserver.HStore(1912): f8160290180ba126c904acc32b60eac7/B is initiating minor compaction (all files) 2023-04-15 02:54:56,100 INFO [RS:0;jenkins-hbase20:34681-longCompactions-0] regionserver.HRegion(2259): Starting compaction of f8160290180ba126c904acc32b60eac7/B in TestAcidGuarantees,,1681527274083.f8160290180ba126c904acc32b60eac7. 2023-04-15 02:54:56,100 INFO [RS:0;jenkins-hbase20:34681-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/f8160290180ba126c904acc32b60eac7/B/f2c8ba28620940748c3055fb37f112be, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/f8160290180ba126c904acc32b60eac7/B/ef70ec718f9246668a85fbea43b0985b, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/f8160290180ba126c904acc32b60eac7/B/4b316212595c42cdb3390715265e9446, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/f8160290180ba126c904acc32b60eac7/B/6c158891089b4e7491d4f81163358b6e] into tmpdir=hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/f8160290180ba126c904acc32b60eac7/.tmp, totalSize=48.4 K 2023-04-15 02:54:56,100 DEBUG [RS:0;jenkins-hbase20:34681-shortCompactions-0] compactions.Compactor(207): Compacting b86d1b177a2d4ecfb61857597a644ad0, keycount=150, bloomtype=ROW, size=12.7 K, encoding=NONE, compression=NONE, seqNum=505, earliestPutTs=1681527291592 2023-04-15 02:54:56,100 DEBUG [RS:0;jenkins-hbase20:34681-longCompactions-0] compactions.Compactor(207): Compacting f2c8ba28620940748c3055fb37f112be, keycount=150, bloomtype=ROW, size=12.7 K, encoding=NONE, compression=NONE, seqNum=505, earliestPutTs=1681527291592 2023-04-15 02:54:56,100 DEBUG [RS:0;jenkins-hbase20:34681-shortCompactions-0] compactions.Compactor(207): Compacting 93b4a56e01ad4055853465b9cbe01d96, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=543, earliestPutTs=1681527293821 2023-04-15 02:54:56,100 DEBUG [RS:0;jenkins-hbase20:34681-longCompactions-0] compactions.Compactor(207): Compacting ef70ec718f9246668a85fbea43b0985b, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=543, earliestPutTs=1681527293821 2023-04-15 02:54:56,101 DEBUG [RS:0;jenkins-hbase20:34681-shortCompactions-0] compactions.Compactor(207): Compacting 23a2c20460fa4566a862fab3caa628ad, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=555, earliestPutTs=1681527294953 2023-04-15 02:54:56,101 DEBUG [RS:0;jenkins-hbase20:34681-longCompactions-0] compactions.Compactor(207): Compacting 4b316212595c42cdb3390715265e9446, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=555, earliestPutTs=1681527294953 2023-04-15 02:54:56,101 DEBUG [RS:0;jenkins-hbase20:34681-shortCompactions-0] compactions.Compactor(207): Compacting c646a2bd66f74fe08577384895aa253b, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=565, earliestPutTs=1681527295579 2023-04-15 02:54:56,101 DEBUG [RS:0;jenkins-hbase20:34681-longCompactions-0] compactions.Compactor(207): Compacting 6c158891089b4e7491d4f81163358b6e, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=565, earliestPutTs=1681527295579 2023-04-15 02:54:56,116 DEBUG [Thread-1527] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-15 02:54:56,117 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-15 02:54:56,121 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-15 02:54:56,131 INFO [RS:0;jenkins-hbase20:34681-longCompactions-0] throttle.PressureAwareThroughputController(145): f8160290180ba126c904acc32b60eac7#B#compaction#266 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-15 02:54:56,137 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-15 02:54:56,137 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-15 02:54:56,137 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-15 02:54:56,137 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-15 02:54:56,137 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-15 02:54:56,137 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-15 02:54:56,137 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:56,138 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:54:56,138 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-15 02:54:56,138 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:56,138 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-15 02:54:56,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-15 02:54:56,139 INFO [RS:0;jenkins-hbase20:34681-shortCompactions-0] throttle.PressureAwareThroughputController(145): f8160290180ba126c904acc32b60eac7#A#compaction#267 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-15 02:54:56,139 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:56,139 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-15 02:54:56,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:54:56,139 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:56,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-15 02:54:56,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:54:56,139 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-15 02:54:56,146 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-15 02:54:56,146 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-15 02:54:56,146 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-15 02:54:56,146 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-15 02:54:56,146 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-15 02:54:56,166 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-15 02:54:56,166 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-15 02:54:56,178 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681527274083.f8160290180ba126c904acc32b60eac7. 2023-04-15 02:54:56,178 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681527274083.f8160290180ba126c904acc32b60eac7. started... 2023-04-15 02:54:56,178 INFO [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing f8160290180ba126c904acc32b60eac7 3/3 column families, dataSize=134.18 KB heapSize=384.98 KB 2023-04-15 02:54:56,178 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK f8160290180ba126c904acc32b60eac7, store=A 2023-04-15 02:54:56,179 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-15 02:54:56,179 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK f8160290180ba126c904acc32b60eac7, store=B 2023-04-15 02:54:56,179 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-15 02:54:56,179 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK f8160290180ba126c904acc32b60eac7, store=C 2023-04-15 02:54:56,179 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-15 02:54:56,217 DEBUG [RS:0;jenkins-hbase20:34681-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/f8160290180ba126c904acc32b60eac7/.tmp/A/371895e17d3c491481f5336a21a159e6 as hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/f8160290180ba126c904acc32b60eac7/A/371895e17d3c491481f5336a21a159e6 2023-04-15 02:54:56,228 DEBUG [RS:0;jenkins-hbase20:34681-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/f8160290180ba126c904acc32b60eac7/.tmp/B/3d5089a21ac3492b8ab691a71b9f7ba3 as hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/f8160290180ba126c904acc32b60eac7/B/3d5089a21ac3492b8ab691a71b9f7ba3 2023-04-15 02:54:56,257 INFO [RS:0;jenkins-hbase20:34681-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in f8160290180ba126c904acc32b60eac7/B of f8160290180ba126c904acc32b60eac7 into 3d5089a21ac3492b8ab691a71b9f7ba3(size=12.9 K), total size for store is 12.9 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-15 02:54:56,257 INFO [RS:0;jenkins-hbase20:34681-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in f8160290180ba126c904acc32b60eac7/A of f8160290180ba126c904acc32b60eac7 into 371895e17d3c491481f5336a21a159e6(size=12.9 K), total size for store is 12.9 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-15 02:54:56,257 DEBUG [RS:0;jenkins-hbase20:34681-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for f8160290180ba126c904acc32b60eac7: 2023-04-15 02:54:56,257 DEBUG [RS:0;jenkins-hbase20:34681-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for f8160290180ba126c904acc32b60eac7: 2023-04-15 02:54:56,257 INFO [RS:0;jenkins-hbase20:34681-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681527274083.f8160290180ba126c904acc32b60eac7., storeName=f8160290180ba126c904acc32b60eac7/B, priority=12, startTime=1681527296096; duration=0sec 2023-04-15 02:54:56,257 INFO [RS:0;jenkins-hbase20:34681-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681527274083.f8160290180ba126c904acc32b60eac7., storeName=f8160290180ba126c904acc32b60eac7/A, priority=12, startTime=1681527296096; duration=0sec 2023-04-15 02:54:56,258 DEBUG [RS:0;jenkins-hbase20:34681-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-15 02:54:56,258 DEBUG [RS:0;jenkins-hbase20:34681-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-15 02:54:56,258 DEBUG [RS:0;jenkins-hbase20:34681-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-15 02:54:56,259 DEBUG [RS:0;jenkins-hbase20:34681-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 49596 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-15 02:54:56,259 DEBUG [RS:0;jenkins-hbase20:34681-longCompactions-0] regionserver.HStore(1912): f8160290180ba126c904acc32b60eac7/C is initiating minor compaction (all files) 2023-04-15 02:54:56,259 INFO [RS:0;jenkins-hbase20:34681-longCompactions-0] regionserver.HRegion(2259): Starting compaction of f8160290180ba126c904acc32b60eac7/C in TestAcidGuarantees,,1681527274083.f8160290180ba126c904acc32b60eac7. 2023-04-15 02:54:56,259 INFO [RS:0;jenkins-hbase20:34681-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/f8160290180ba126c904acc32b60eac7/C/49468c20795140b7bd23c7e29c1bd3e2, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/f8160290180ba126c904acc32b60eac7/C/f64cc99821ad4f14926e330d9a735d6b, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/f8160290180ba126c904acc32b60eac7/C/619b1f4e5ede48f18c9511b08d2d11f8, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/f8160290180ba126c904acc32b60eac7/C/ecc6d46705bd4be588a7418471c777ec] into tmpdir=hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/f8160290180ba126c904acc32b60eac7/.tmp, totalSize=48.4 K 2023-04-15 02:54:56,259 DEBUG [RS:0;jenkins-hbase20:34681-longCompactions-0] compactions.Compactor(207): Compacting 49468c20795140b7bd23c7e29c1bd3e2, keycount=150, bloomtype=ROW, size=12.7 K, encoding=NONE, compression=NONE, seqNum=505, earliestPutTs=1681527291592 2023-04-15 02:54:56,259 DEBUG [RS:0;jenkins-hbase20:34681-longCompactions-0] compactions.Compactor(207): Compacting f64cc99821ad4f14926e330d9a735d6b, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=543, earliestPutTs=1681527293821 2023-04-15 02:54:56,260 DEBUG [RS:0;jenkins-hbase20:34681-longCompactions-0] compactions.Compactor(207): Compacting 619b1f4e5ede48f18c9511b08d2d11f8, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=555, earliestPutTs=1681527294953 2023-04-15 02:54:56,260 DEBUG [RS:0;jenkins-hbase20:34681-longCompactions-0] compactions.Compactor(207): Compacting ecc6d46705bd4be588a7418471c777ec, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=565, earliestPutTs=1681527295579 2023-04-15 02:54:56,267 INFO [RS:0;jenkins-hbase20:34681-longCompactions-0] throttle.PressureAwareThroughputController(145): f8160290180ba126c904acc32b60eac7#C#compaction#269 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-15 02:54:56,279 DEBUG [Thread-1534] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x4cbc74e7 to 127.0.0.1:51762 2023-04-15 02:54:56,279 DEBUG [Thread-1534] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-15 02:54:56,280 DEBUG [Thread-1536] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x3c5d1710 to 127.0.0.1:51762 2023-04-15 02:54:56,280 DEBUG [Thread-1536] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-15 02:54:56,280 DEBUG [Thread-1530] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x6b7f716c to 127.0.0.1:51762 2023-04-15 02:54:56,280 DEBUG [Thread-1530] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-15 02:54:56,281 DEBUG [Thread-1532] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x6ade356f to 127.0.0.1:51762 2023-04-15 02:54:56,281 DEBUG [Thread-1532] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-15 02:54:56,281 DEBUG [Thread-1528] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x7e3efec6 to 127.0.0.1:51762 2023-04-15 02:54:56,281 DEBUG [Thread-1528] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-15 02:54:56,333 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34681] regionserver.HRegion(9158): Flush requested on f8160290180ba126c904acc32b60eac7 2023-04-15 02:54:56,333 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681527274083.f8160290180ba126c904acc32b60eac7. as already flushing 2023-04-15 02:54:56,333 DEBUG [Thread-1519] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x7117e4df to 127.0.0.1:51762 2023-04-15 02:54:56,333 DEBUG [Thread-1519] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-15 02:54:56,665 INFO [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=602 (bloomFilter=true), to=hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/f8160290180ba126c904acc32b60eac7/.tmp/A/e41d6a86a022461ca5b1042b208c8158 2023-04-15 02:54:56,692 DEBUG [RS:0;jenkins-hbase20:34681-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/f8160290180ba126c904acc32b60eac7/.tmp/C/9cc512d9db0d48789e4ecb33dfb397fe as hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/f8160290180ba126c904acc32b60eac7/C/9cc512d9db0d48789e4ecb33dfb397fe 2023-04-15 02:54:56,697 INFO [RS:0;jenkins-hbase20:34681-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in f8160290180ba126c904acc32b60eac7/C of f8160290180ba126c904acc32b60eac7 into 9cc512d9db0d48789e4ecb33dfb397fe(size=12.9 K), total size for store is 12.9 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-15 02:54:56,697 DEBUG [RS:0;jenkins-hbase20:34681-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for f8160290180ba126c904acc32b60eac7: 2023-04-15 02:54:56,697 INFO [RS:0;jenkins-hbase20:34681-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681527274083.f8160290180ba126c904acc32b60eac7., storeName=f8160290180ba126c904acc32b60eac7/C, priority=12, startTime=1681527296097; duration=0sec 2023-04-15 02:54:56,697 DEBUG [RS:0;jenkins-hbase20:34681-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-15 02:54:57,076 INFO [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=602 (bloomFilter=true), to=hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/f8160290180ba126c904acc32b60eac7/.tmp/B/43de67ec7bb1430ba0ea05932a731fd1 2023-04-15 02:54:57,489 INFO [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=602 (bloomFilter=true), to=hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/f8160290180ba126c904acc32b60eac7/.tmp/C/ccb42311a7bb487cad982f7836f0e907 2023-04-15 02:54:57,492 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/f8160290180ba126c904acc32b60eac7/.tmp/A/e41d6a86a022461ca5b1042b208c8158 as hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/f8160290180ba126c904acc32b60eac7/A/e41d6a86a022461ca5b1042b208c8158 2023-04-15 02:54:57,496 INFO [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/f8160290180ba126c904acc32b60eac7/A/e41d6a86a022461ca5b1042b208c8158, entries=150, sequenceid=602, filesize=11.9 K 2023-04-15 02:54:57,497 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/f8160290180ba126c904acc32b60eac7/.tmp/B/43de67ec7bb1430ba0ea05932a731fd1 as hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/f8160290180ba126c904acc32b60eac7/B/43de67ec7bb1430ba0ea05932a731fd1 2023-04-15 02:54:57,500 INFO [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/f8160290180ba126c904acc32b60eac7/B/43de67ec7bb1430ba0ea05932a731fd1, entries=150, sequenceid=602, filesize=11.9 K 2023-04-15 02:54:57,501 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/f8160290180ba126c904acc32b60eac7/.tmp/C/ccb42311a7bb487cad982f7836f0e907 as hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/f8160290180ba126c904acc32b60eac7/C/ccb42311a7bb487cad982f7836f0e907 2023-04-15 02:54:57,509 INFO [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/f8160290180ba126c904acc32b60eac7/C/ccb42311a7bb487cad982f7836f0e907, entries=150, sequenceid=602, filesize=11.9 K 2023-04-15 02:54:57,509 INFO [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~134.18 KB/137400, heapSize ~384.94 KB/394176, currentSize=6.71 KB/6870 for f8160290180ba126c904acc32b60eac7 in 1331ms, sequenceid=602, compaction requested=false 2023-04-15 02:54:57,509 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for f8160290180ba126c904acc32b60eac7: 2023-04-15 02:54:57,509 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681527274083.f8160290180ba126c904acc32b60eac7. 2023-04-15 02:54:57,509 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-15 02:54:57,510 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-15 02:54:57,510 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:57,510 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-15 02:54:57,510 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34681,1681527209133' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-15 02:54:57,511 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:57,511 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:57,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:57,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:54:57,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:54:57,511 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:57,511 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-15 02:54:57,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:54:57,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:54:57,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:54:57,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:57,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:54:57,512 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34681,1681527209133' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-15 02:54:57,512 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@18a68004[Count = 0] remaining members to acquire global barrier 2023-04-15 02:54:57,512 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-15 02:54:57,512 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:57,513 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:57,513 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:57,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:57,513 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-15 02:54:57,513 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-15 02:54:57,513 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:57,513 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-15 02:54:57,513 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34681,1681527209133' in zk 2023-04-15 02:54:57,514 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:57,514 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-15 02:54:57,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:57,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:54:57,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:54:57,514 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-15 02:54:57,514 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-15 02:54:57,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:54:57,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:54:57,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:54:57,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:57,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:54:57,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:54:57,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:57,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34681,1681527209133': 2023-04-15 02:54:57,516 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34681,1681527209133' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-15 02:54:57,516 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-15 02:54:57,516 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-15 02:54:57,516 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-15 02:54:57,516 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-15 02:54:57,517 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-15 02:54:57,518 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:57,518 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:57,518 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:57,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:57,518 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-15 02:54:57,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:57,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:54:57,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:54:57,518 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-15 02:54:57,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-15 02:54:57,518 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:57,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:54:57,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:57,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:54:57,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:54:57,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:54:57,520 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:57,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:57,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:54:57,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:54:57,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:57,521 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:57,521 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:54:57,521 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:54:57,521 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:57,521 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:57,521 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-15 02:54:57,521 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-15 02:54:57,521 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-15 02:54:57,521 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-15 02:54:57,522 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-15 02:54:57,522 DEBUG [Thread-1527] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-15 02:54:57,522 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-15 02:54:57,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-15 02:54:57,522 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-15 02:54:57,521 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:54:57,522 DEBUG [Thread-1527] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-15 02:54:57,522 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:54:57,522 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-15 02:54:57,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-15 02:54:57,622 DEBUG [Thread-1527] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-15 02:54:57,623 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-15 02:54:59,614 DEBUG [Thread-1521] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x5c466b6c to 127.0.0.1:51762 2023-04-15 02:54:59,614 DEBUG [Thread-1521] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-15 02:54:59,680 DEBUG [Thread-1523] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x46d74611 to 127.0.0.1:51762 2023-04-15 02:54:59,680 DEBUG [Thread-1523] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-15 02:54:59,697 DEBUG [Thread-1525] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x3f837278 to 127.0.0.1:51762 2023-04-15 02:54:59,698 DEBUG [Thread-1525] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-15 02:54:59,699 DEBUG [Thread-1517] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x64e0df67 to 127.0.0.1:51762 2023-04-15 02:54:59,699 DEBUG [Thread-1517] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-15 02:54:59,699 INFO [Listener at localhost.localdomain/33693] hbase.AcidGuaranteesTestTool(392): Finished test. Writers: 2023-04-15 02:54:59,699 INFO [Listener at localhost.localdomain/33693] hbase.AcidGuaranteesTestTool(394): wrote 32 2023-04-15 02:54:59,699 INFO [Listener at localhost.localdomain/33693] hbase.AcidGuaranteesTestTool(394): wrote 404 2023-04-15 02:54:59,700 INFO [Listener at localhost.localdomain/33693] hbase.AcidGuaranteesTestTool(394): wrote 19 2023-04-15 02:54:59,700 INFO [Listener at localhost.localdomain/33693] hbase.AcidGuaranteesTestTool(394): wrote 20 2023-04-15 02:54:59,700 INFO [Listener at localhost.localdomain/33693] hbase.AcidGuaranteesTestTool(394): wrote 36 2023-04-15 02:54:59,700 INFO [Listener at localhost.localdomain/33693] hbase.AcidGuaranteesTestTool(396): Readers: 2023-04-15 02:54:59,700 INFO [Listener at localhost.localdomain/33693] hbase.AcidGuaranteesTestTool(398): read 5294 2023-04-15 02:54:59,700 INFO [Listener at localhost.localdomain/33693] hbase.AcidGuaranteesTestTool(398): read 5354 2023-04-15 02:54:59,700 INFO [Listener at localhost.localdomain/33693] hbase.AcidGuaranteesTestTool(398): read 5024 2023-04-15 02:54:59,700 INFO [Listener at localhost.localdomain/33693] hbase.AcidGuaranteesTestTool(398): read 5284 2023-04-15 02:54:59,700 INFO [Listener at localhost.localdomain/33693] hbase.AcidGuaranteesTestTool(398): read 5309 2023-04-15 02:54:59,700 INFO [Listener at localhost.localdomain/33693] hbase.AcidGuaranteesTestTool(400): Scanners: 2023-04-15 02:54:59,700 INFO [Listener at localhost.localdomain/33693] client.ConnectionImplementation(1974): Closing master protocol: MasterService 2023-04-15 02:54:59,700 DEBUG [Listener at localhost.localdomain/33693] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x29efb2d8 to 127.0.0.1:51762 2023-04-15 02:54:59,700 DEBUG [Listener at localhost.localdomain/33693] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-15 02:54:59,701 INFO [Listener at localhost.localdomain/33693] client.HBaseAdmin$15(890): Started disable of TestAcidGuarantees 2023-04-15 02:54:59,701 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.HMaster$11(2418): Client=jenkins//148.251.75.209 disable TestAcidGuarantees 2023-04-15 02:54:59,701 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] procedure2.ProcedureExecutor(1029): Stored pid=31, state=RUNNABLE:DISABLE_TABLE_PREPARE; DisableTableProcedure table=TestAcidGuarantees 2023-04-15 02:54:59,703 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(1227): Checking to see if procedure is done pid=31 2023-04-15 02:54:59,704 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1681527299704"}]},"ts":"1681527299704"} 2023-04-15 02:54:59,704 INFO [PEWorker-5] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=DISABLING in hbase:meta 2023-04-15 02:54:59,705 INFO [PEWorker-5] procedure.DisableTableProcedure(293): Set TestAcidGuarantees to state=DISABLING 2023-04-15 02:54:59,706 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=32, ppid=31, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=f8160290180ba126c904acc32b60eac7, UNASSIGN}] 2023-04-15 02:54:59,707 INFO [PEWorker-4] procedure.MasterProcedureScheduler(727): Took xlock for pid=32, ppid=31, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=f8160290180ba126c904acc32b60eac7, UNASSIGN 2023-04-15 02:54:59,708 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=32 updating hbase:meta row=f8160290180ba126c904acc32b60eac7, regionState=CLOSING, regionLocation=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:54:59,708 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1681527274083.f8160290180ba126c904acc32b60eac7.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1681527299708"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1681527299708"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1681527299708"}]},"ts":"1681527299708"} 2023-04-15 02:54:59,709 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=33, ppid=32, state=RUNNABLE; CloseRegionProcedure f8160290180ba126c904acc32b60eac7, server=jenkins-hbase20.apache.org,34681,1681527209133}] 2023-04-15 02:54:59,804 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(1227): Checking to see if procedure is done pid=31 2023-04-15 02:54:59,861 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] handler.UnassignRegionHandler(109): Close f8160290180ba126c904acc32b60eac7 2023-04-15 02:54:59,861 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1604): Closing f8160290180ba126c904acc32b60eac7, disabling compactions & flushes 2023-04-15 02:54:59,861 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1681527274083.f8160290180ba126c904acc32b60eac7. 2023-04-15 02:54:59,861 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1681527274083.f8160290180ba126c904acc32b60eac7. 2023-04-15 02:54:59,861 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1681527274083.f8160290180ba126c904acc32b60eac7. after waiting 0 ms 2023-04-15 02:54:59,861 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1681527274083.f8160290180ba126c904acc32b60eac7. 2023-04-15 02:54:59,861 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(2745): Flushing f8160290180ba126c904acc32b60eac7 3/3 column families, dataSize=33.54 KB heapSize=100.36 KB 2023-04-15 02:54:59,861 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK f8160290180ba126c904acc32b60eac7, store=A 2023-04-15 02:54:59,861 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-15 02:54:59,862 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK f8160290180ba126c904acc32b60eac7, store=B 2023-04-15 02:54:59,862 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-15 02:54:59,862 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK f8160290180ba126c904acc32b60eac7, store=C 2023-04-15 02:54:59,862 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-15 02:54:59,869 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=613 (bloomFilter=true), to=hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/f8160290180ba126c904acc32b60eac7/.tmp/A/818405efe3054b6c80231b64e1ed28b1 2023-04-15 02:55:00,005 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(1227): Checking to see if procedure is done pid=31 2023-04-15 02:55:00,278 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=613 (bloomFilter=true), to=hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/f8160290180ba126c904acc32b60eac7/.tmp/B/ed1b2246733d494382852c282b1727b0 2023-04-15 02:55:00,306 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(1227): Checking to see if procedure is done pid=31 2023-04-15 02:55:00,690 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=613 (bloomFilter=true), to=hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/f8160290180ba126c904acc32b60eac7/.tmp/C/20f1b0d0f922450b8088c359da4dc910 2023-04-15 02:55:00,695 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/f8160290180ba126c904acc32b60eac7/.tmp/A/818405efe3054b6c80231b64e1ed28b1 as hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/f8160290180ba126c904acc32b60eac7/A/818405efe3054b6c80231b64e1ed28b1 2023-04-15 02:55:00,698 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/f8160290180ba126c904acc32b60eac7/A/818405efe3054b6c80231b64e1ed28b1, entries=150, sequenceid=613, filesize=11.9 K 2023-04-15 02:55:00,699 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/f8160290180ba126c904acc32b60eac7/.tmp/B/ed1b2246733d494382852c282b1727b0 as hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/f8160290180ba126c904acc32b60eac7/B/ed1b2246733d494382852c282b1727b0 2023-04-15 02:55:00,703 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/f8160290180ba126c904acc32b60eac7/B/ed1b2246733d494382852c282b1727b0, entries=150, sequenceid=613, filesize=11.9 K 2023-04-15 02:55:00,704 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/f8160290180ba126c904acc32b60eac7/.tmp/C/20f1b0d0f922450b8088c359da4dc910 as hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/f8160290180ba126c904acc32b60eac7/C/20f1b0d0f922450b8088c359da4dc910 2023-04-15 02:55:00,709 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/f8160290180ba126c904acc32b60eac7/C/20f1b0d0f922450b8088c359da4dc910, entries=150, sequenceid=613, filesize=11.9 K 2023-04-15 02:55:00,710 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(2948): Finished flush of dataSize ~33.54 KB/34350, heapSize ~100.31 KB/102720, currentSize=0 B/0 for f8160290180ba126c904acc32b60eac7 in 849ms, sequenceid=613, compaction requested=true 2023-04-15 02:55:00,738 DEBUG [StoreCloser-TestAcidGuarantees,,1681527274083.f8160290180ba126c904acc32b60eac7.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/f8160290180ba126c904acc32b60eac7/A/40f07466e76f415ab019342c8727995e, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/f8160290180ba126c904acc32b60eac7/A/9d601bd6d01f4a3f808814012e1a1b12, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/f8160290180ba126c904acc32b60eac7/A/f4e10dcc69934f22941dd7fde1d40ee1, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/f8160290180ba126c904acc32b60eac7/A/86620e56e72f459a99b0d456ff3d5e39, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/f8160290180ba126c904acc32b60eac7/A/61dcb71e9705420ab52f98869f5eab5a, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/f8160290180ba126c904acc32b60eac7/A/0944ea0d48e048fe8486e8aa032e1470, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/f8160290180ba126c904acc32b60eac7/A/3252d9d39f894e3e98cd1a5aa3e3014d, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/f8160290180ba126c904acc32b60eac7/A/b562b2c57098424c807c032a98e2362e, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/f8160290180ba126c904acc32b60eac7/A/f9cf581ca3df4acdad0e1c123fb09953, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/f8160290180ba126c904acc32b60eac7/A/7d6f907648d94f078af742e9bed125b2, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/f8160290180ba126c904acc32b60eac7/A/b8354584db9a4f43b0b4e9eef47e1892, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/f8160290180ba126c904acc32b60eac7/A/954aa6d0069848788bf78dade2b6f2e7, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/f8160290180ba126c904acc32b60eac7/A/6846a5a1400b41fb908682d63eb5d93e, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/f8160290180ba126c904acc32b60eac7/A/9276f313bec440ab82008fb21fe3271a, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/f8160290180ba126c904acc32b60eac7/A/774c69fb42ef48febac3611cacd2e05f, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/f8160290180ba126c904acc32b60eac7/A/4c132a56d87d4a4089c66bc0225f2389, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/f8160290180ba126c904acc32b60eac7/A/e73d4d3b96aa40cd914bc600fa1a7979, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/f8160290180ba126c904acc32b60eac7/A/0c0af3ff076a46db963ed21c3b7e3d42, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/f8160290180ba126c904acc32b60eac7/A/ed9f1352085e4cd4a6046951a999c3fb, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/f8160290180ba126c904acc32b60eac7/A/8ec77327a35847be8ff2bba511070406, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/f8160290180ba126c904acc32b60eac7/A/88d467e58ea44e1fbf4749826d79cc70, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/f8160290180ba126c904acc32b60eac7/A/9004437d3cc24f318e2e9d2d5d21a388, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/f8160290180ba126c904acc32b60eac7/A/1f24a205b89340e69eadd12739c7976f, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/f8160290180ba126c904acc32b60eac7/A/465c74a754914f38b43b1f7bbea09d26, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/f8160290180ba126c904acc32b60eac7/A/b86d1b177a2d4ecfb61857597a644ad0, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/f8160290180ba126c904acc32b60eac7/A/8cc3365625a74bbe9ffca4472e1584e8, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/f8160290180ba126c904acc32b60eac7/A/93b4a56e01ad4055853465b9cbe01d96, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/f8160290180ba126c904acc32b60eac7/A/23a2c20460fa4566a862fab3caa628ad, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/f8160290180ba126c904acc32b60eac7/A/c646a2bd66f74fe08577384895aa253b] to archive 2023-04-15 02:55:00,739 DEBUG [StoreCloser-TestAcidGuarantees,,1681527274083.f8160290180ba126c904acc32b60eac7.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-15 02:55:00,741 DEBUG [StoreCloser-TestAcidGuarantees,,1681527274083.f8160290180ba126c904acc32b60eac7.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/f8160290180ba126c904acc32b60eac7/A/40f07466e76f415ab019342c8727995e to hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/archive/data/default/TestAcidGuarantees/f8160290180ba126c904acc32b60eac7/A/40f07466e76f415ab019342c8727995e 2023-04-15 02:55:00,742 DEBUG [StoreCloser-TestAcidGuarantees,,1681527274083.f8160290180ba126c904acc32b60eac7.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/f8160290180ba126c904acc32b60eac7/A/9d601bd6d01f4a3f808814012e1a1b12 to hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/archive/data/default/TestAcidGuarantees/f8160290180ba126c904acc32b60eac7/A/9d601bd6d01f4a3f808814012e1a1b12 2023-04-15 02:55:00,744 DEBUG [StoreCloser-TestAcidGuarantees,,1681527274083.f8160290180ba126c904acc32b60eac7.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/f8160290180ba126c904acc32b60eac7/A/f4e10dcc69934f22941dd7fde1d40ee1 to hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/archive/data/default/TestAcidGuarantees/f8160290180ba126c904acc32b60eac7/A/f4e10dcc69934f22941dd7fde1d40ee1 2023-04-15 02:55:00,746 DEBUG [StoreCloser-TestAcidGuarantees,,1681527274083.f8160290180ba126c904acc32b60eac7.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/f8160290180ba126c904acc32b60eac7/A/86620e56e72f459a99b0d456ff3d5e39 to hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/archive/data/default/TestAcidGuarantees/f8160290180ba126c904acc32b60eac7/A/86620e56e72f459a99b0d456ff3d5e39 2023-04-15 02:55:00,747 DEBUG [StoreCloser-TestAcidGuarantees,,1681527274083.f8160290180ba126c904acc32b60eac7.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/f8160290180ba126c904acc32b60eac7/A/61dcb71e9705420ab52f98869f5eab5a to hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/archive/data/default/TestAcidGuarantees/f8160290180ba126c904acc32b60eac7/A/61dcb71e9705420ab52f98869f5eab5a 2023-04-15 02:55:00,749 DEBUG [StoreCloser-TestAcidGuarantees,,1681527274083.f8160290180ba126c904acc32b60eac7.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/f8160290180ba126c904acc32b60eac7/A/0944ea0d48e048fe8486e8aa032e1470 to hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/archive/data/default/TestAcidGuarantees/f8160290180ba126c904acc32b60eac7/A/0944ea0d48e048fe8486e8aa032e1470 2023-04-15 02:55:00,750 DEBUG [StoreCloser-TestAcidGuarantees,,1681527274083.f8160290180ba126c904acc32b60eac7.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/f8160290180ba126c904acc32b60eac7/A/3252d9d39f894e3e98cd1a5aa3e3014d to hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/archive/data/default/TestAcidGuarantees/f8160290180ba126c904acc32b60eac7/A/3252d9d39f894e3e98cd1a5aa3e3014d 2023-04-15 02:55:00,751 DEBUG [StoreCloser-TestAcidGuarantees,,1681527274083.f8160290180ba126c904acc32b60eac7.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/f8160290180ba126c904acc32b60eac7/A/b562b2c57098424c807c032a98e2362e to hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/archive/data/default/TestAcidGuarantees/f8160290180ba126c904acc32b60eac7/A/b562b2c57098424c807c032a98e2362e 2023-04-15 02:55:00,752 DEBUG [StoreCloser-TestAcidGuarantees,,1681527274083.f8160290180ba126c904acc32b60eac7.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/f8160290180ba126c904acc32b60eac7/A/f9cf581ca3df4acdad0e1c123fb09953 to hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/archive/data/default/TestAcidGuarantees/f8160290180ba126c904acc32b60eac7/A/f9cf581ca3df4acdad0e1c123fb09953 2023-04-15 02:55:00,753 DEBUG [StoreCloser-TestAcidGuarantees,,1681527274083.f8160290180ba126c904acc32b60eac7.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/f8160290180ba126c904acc32b60eac7/A/7d6f907648d94f078af742e9bed125b2 to hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/archive/data/default/TestAcidGuarantees/f8160290180ba126c904acc32b60eac7/A/7d6f907648d94f078af742e9bed125b2 2023-04-15 02:55:00,754 DEBUG [StoreCloser-TestAcidGuarantees,,1681527274083.f8160290180ba126c904acc32b60eac7.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/f8160290180ba126c904acc32b60eac7/A/b8354584db9a4f43b0b4e9eef47e1892 to hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/archive/data/default/TestAcidGuarantees/f8160290180ba126c904acc32b60eac7/A/b8354584db9a4f43b0b4e9eef47e1892 2023-04-15 02:55:00,755 DEBUG [StoreCloser-TestAcidGuarantees,,1681527274083.f8160290180ba126c904acc32b60eac7.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/f8160290180ba126c904acc32b60eac7/A/954aa6d0069848788bf78dade2b6f2e7 to hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/archive/data/default/TestAcidGuarantees/f8160290180ba126c904acc32b60eac7/A/954aa6d0069848788bf78dade2b6f2e7 2023-04-15 02:55:00,756 DEBUG [StoreCloser-TestAcidGuarantees,,1681527274083.f8160290180ba126c904acc32b60eac7.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/f8160290180ba126c904acc32b60eac7/A/6846a5a1400b41fb908682d63eb5d93e to hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/archive/data/default/TestAcidGuarantees/f8160290180ba126c904acc32b60eac7/A/6846a5a1400b41fb908682d63eb5d93e 2023-04-15 02:55:00,757 DEBUG [StoreCloser-TestAcidGuarantees,,1681527274083.f8160290180ba126c904acc32b60eac7.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/f8160290180ba126c904acc32b60eac7/A/9276f313bec440ab82008fb21fe3271a to hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/archive/data/default/TestAcidGuarantees/f8160290180ba126c904acc32b60eac7/A/9276f313bec440ab82008fb21fe3271a 2023-04-15 02:55:00,757 DEBUG [StoreCloser-TestAcidGuarantees,,1681527274083.f8160290180ba126c904acc32b60eac7.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/f8160290180ba126c904acc32b60eac7/A/774c69fb42ef48febac3611cacd2e05f to hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/archive/data/default/TestAcidGuarantees/f8160290180ba126c904acc32b60eac7/A/774c69fb42ef48febac3611cacd2e05f 2023-04-15 02:55:00,758 DEBUG [StoreCloser-TestAcidGuarantees,,1681527274083.f8160290180ba126c904acc32b60eac7.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/f8160290180ba126c904acc32b60eac7/A/4c132a56d87d4a4089c66bc0225f2389 to hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/archive/data/default/TestAcidGuarantees/f8160290180ba126c904acc32b60eac7/A/4c132a56d87d4a4089c66bc0225f2389 2023-04-15 02:55:00,759 DEBUG [StoreCloser-TestAcidGuarantees,,1681527274083.f8160290180ba126c904acc32b60eac7.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/f8160290180ba126c904acc32b60eac7/A/e73d4d3b96aa40cd914bc600fa1a7979 to hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/archive/data/default/TestAcidGuarantees/f8160290180ba126c904acc32b60eac7/A/e73d4d3b96aa40cd914bc600fa1a7979 2023-04-15 02:55:00,760 DEBUG [StoreCloser-TestAcidGuarantees,,1681527274083.f8160290180ba126c904acc32b60eac7.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/f8160290180ba126c904acc32b60eac7/A/0c0af3ff076a46db963ed21c3b7e3d42 to hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/archive/data/default/TestAcidGuarantees/f8160290180ba126c904acc32b60eac7/A/0c0af3ff076a46db963ed21c3b7e3d42 2023-04-15 02:55:00,761 DEBUG [StoreCloser-TestAcidGuarantees,,1681527274083.f8160290180ba126c904acc32b60eac7.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/f8160290180ba126c904acc32b60eac7/A/ed9f1352085e4cd4a6046951a999c3fb to hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/archive/data/default/TestAcidGuarantees/f8160290180ba126c904acc32b60eac7/A/ed9f1352085e4cd4a6046951a999c3fb 2023-04-15 02:55:00,762 DEBUG [StoreCloser-TestAcidGuarantees,,1681527274083.f8160290180ba126c904acc32b60eac7.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/f8160290180ba126c904acc32b60eac7/A/8ec77327a35847be8ff2bba511070406 to hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/archive/data/default/TestAcidGuarantees/f8160290180ba126c904acc32b60eac7/A/8ec77327a35847be8ff2bba511070406 2023-04-15 02:55:00,763 DEBUG [StoreCloser-TestAcidGuarantees,,1681527274083.f8160290180ba126c904acc32b60eac7.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/f8160290180ba126c904acc32b60eac7/A/88d467e58ea44e1fbf4749826d79cc70 to hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/archive/data/default/TestAcidGuarantees/f8160290180ba126c904acc32b60eac7/A/88d467e58ea44e1fbf4749826d79cc70 2023-04-15 02:55:00,764 DEBUG [StoreCloser-TestAcidGuarantees,,1681527274083.f8160290180ba126c904acc32b60eac7.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/f8160290180ba126c904acc32b60eac7/A/9004437d3cc24f318e2e9d2d5d21a388 to hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/archive/data/default/TestAcidGuarantees/f8160290180ba126c904acc32b60eac7/A/9004437d3cc24f318e2e9d2d5d21a388 2023-04-15 02:55:00,766 DEBUG [StoreCloser-TestAcidGuarantees,,1681527274083.f8160290180ba126c904acc32b60eac7.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/f8160290180ba126c904acc32b60eac7/A/1f24a205b89340e69eadd12739c7976f to hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/archive/data/default/TestAcidGuarantees/f8160290180ba126c904acc32b60eac7/A/1f24a205b89340e69eadd12739c7976f 2023-04-15 02:55:00,766 DEBUG [StoreCloser-TestAcidGuarantees,,1681527274083.f8160290180ba126c904acc32b60eac7.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/f8160290180ba126c904acc32b60eac7/A/465c74a754914f38b43b1f7bbea09d26 to hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/archive/data/default/TestAcidGuarantees/f8160290180ba126c904acc32b60eac7/A/465c74a754914f38b43b1f7bbea09d26 2023-04-15 02:55:00,767 DEBUG [StoreCloser-TestAcidGuarantees,,1681527274083.f8160290180ba126c904acc32b60eac7.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/f8160290180ba126c904acc32b60eac7/A/b86d1b177a2d4ecfb61857597a644ad0 to hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/archive/data/default/TestAcidGuarantees/f8160290180ba126c904acc32b60eac7/A/b86d1b177a2d4ecfb61857597a644ad0 2023-04-15 02:55:00,768 DEBUG [StoreCloser-TestAcidGuarantees,,1681527274083.f8160290180ba126c904acc32b60eac7.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/f8160290180ba126c904acc32b60eac7/A/8cc3365625a74bbe9ffca4472e1584e8 to hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/archive/data/default/TestAcidGuarantees/f8160290180ba126c904acc32b60eac7/A/8cc3365625a74bbe9ffca4472e1584e8 2023-04-15 02:55:00,769 DEBUG [StoreCloser-TestAcidGuarantees,,1681527274083.f8160290180ba126c904acc32b60eac7.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/f8160290180ba126c904acc32b60eac7/A/93b4a56e01ad4055853465b9cbe01d96 to hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/archive/data/default/TestAcidGuarantees/f8160290180ba126c904acc32b60eac7/A/93b4a56e01ad4055853465b9cbe01d96 2023-04-15 02:55:00,770 DEBUG [StoreCloser-TestAcidGuarantees,,1681527274083.f8160290180ba126c904acc32b60eac7.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/f8160290180ba126c904acc32b60eac7/A/23a2c20460fa4566a862fab3caa628ad to hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/archive/data/default/TestAcidGuarantees/f8160290180ba126c904acc32b60eac7/A/23a2c20460fa4566a862fab3caa628ad 2023-04-15 02:55:00,771 DEBUG [StoreCloser-TestAcidGuarantees,,1681527274083.f8160290180ba126c904acc32b60eac7.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/f8160290180ba126c904acc32b60eac7/A/c646a2bd66f74fe08577384895aa253b to hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/archive/data/default/TestAcidGuarantees/f8160290180ba126c904acc32b60eac7/A/c646a2bd66f74fe08577384895aa253b 2023-04-15 02:55:00,774 DEBUG [StoreCloser-TestAcidGuarantees,,1681527274083.f8160290180ba126c904acc32b60eac7.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/f8160290180ba126c904acc32b60eac7/B/b8c69868004c408eb9d6fd873d117041, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/f8160290180ba126c904acc32b60eac7/B/abc1edeb35f04e01b9b1176e384165f7, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/f8160290180ba126c904acc32b60eac7/B/e8ba002fe0f945c0b9561495f7b6e3a5, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/f8160290180ba126c904acc32b60eac7/B/2e47e14a986e4838b5afda846a0524da, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/f8160290180ba126c904acc32b60eac7/B/0f93ea44f44f4b189b1a30893dbf6367, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/f8160290180ba126c904acc32b60eac7/B/0b2852aa58d9444994a5a57c93b0d068, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/f8160290180ba126c904acc32b60eac7/B/b279733ba1f64aa5af8bfe1a1e7e0cb5, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/f8160290180ba126c904acc32b60eac7/B/ba80e0ee3a2249e79c5e1f0fb611e042, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/f8160290180ba126c904acc32b60eac7/B/1e7004cf2a474345befc018fc8adcf7f, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/f8160290180ba126c904acc32b60eac7/B/fffc50bcd93a4e7e9a049c6eddc9b678, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/f8160290180ba126c904acc32b60eac7/B/2b8f9e91423a47e59fbc1a268696347b, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/f8160290180ba126c904acc32b60eac7/B/70be9726a32642b48c6b33242243860b, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/f8160290180ba126c904acc32b60eac7/B/e8e29ea90fdc4fdeb8353ff44c9293d2, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/f8160290180ba126c904acc32b60eac7/B/f035898cd29a4386814b629068363bef, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/f8160290180ba126c904acc32b60eac7/B/1af10683cc6a452aaece1b4f85a3c87b, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/f8160290180ba126c904acc32b60eac7/B/4eca2f71b8374fbd86068b6b3f1136e8, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/f8160290180ba126c904acc32b60eac7/B/9d7fee285c1d4831abc6577d7be84f9a, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/f8160290180ba126c904acc32b60eac7/B/48c53c106b4642519b29b1e401289ef7, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/f8160290180ba126c904acc32b60eac7/B/a87b02c9f45148c6b5788737b0ebb158, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/f8160290180ba126c904acc32b60eac7/B/cf9b4a5be1174051ab96d62fbd409586, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/f8160290180ba126c904acc32b60eac7/B/ea1cf09ab444475b96e383a56dc20362, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/f8160290180ba126c904acc32b60eac7/B/a9b63f57015d4c7ba5ccf3f0792317a5, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/f8160290180ba126c904acc32b60eac7/B/ee48283915b74aa6b528d9fb9f3189f7, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/f8160290180ba126c904acc32b60eac7/B/219e1fe89c37434394f36af8af7460ec, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/f8160290180ba126c904acc32b60eac7/B/f2c8ba28620940748c3055fb37f112be, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/f8160290180ba126c904acc32b60eac7/B/805958efea2344fdb254357346ad44de, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/f8160290180ba126c904acc32b60eac7/B/ef70ec718f9246668a85fbea43b0985b, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/f8160290180ba126c904acc32b60eac7/B/4b316212595c42cdb3390715265e9446, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/f8160290180ba126c904acc32b60eac7/B/6c158891089b4e7491d4f81163358b6e] to archive 2023-04-15 02:55:00,774 DEBUG [StoreCloser-TestAcidGuarantees,,1681527274083.f8160290180ba126c904acc32b60eac7.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-15 02:55:00,776 DEBUG [StoreCloser-TestAcidGuarantees,,1681527274083.f8160290180ba126c904acc32b60eac7.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/f8160290180ba126c904acc32b60eac7/B/b8c69868004c408eb9d6fd873d117041 to hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/archive/data/default/TestAcidGuarantees/f8160290180ba126c904acc32b60eac7/B/b8c69868004c408eb9d6fd873d117041 2023-04-15 02:55:00,777 DEBUG [StoreCloser-TestAcidGuarantees,,1681527274083.f8160290180ba126c904acc32b60eac7.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/f8160290180ba126c904acc32b60eac7/B/abc1edeb35f04e01b9b1176e384165f7 to hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/archive/data/default/TestAcidGuarantees/f8160290180ba126c904acc32b60eac7/B/abc1edeb35f04e01b9b1176e384165f7 2023-04-15 02:55:00,778 DEBUG [StoreCloser-TestAcidGuarantees,,1681527274083.f8160290180ba126c904acc32b60eac7.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/f8160290180ba126c904acc32b60eac7/B/e8ba002fe0f945c0b9561495f7b6e3a5 to hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/archive/data/default/TestAcidGuarantees/f8160290180ba126c904acc32b60eac7/B/e8ba002fe0f945c0b9561495f7b6e3a5 2023-04-15 02:55:00,779 DEBUG [StoreCloser-TestAcidGuarantees,,1681527274083.f8160290180ba126c904acc32b60eac7.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/f8160290180ba126c904acc32b60eac7/B/2e47e14a986e4838b5afda846a0524da to hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/archive/data/default/TestAcidGuarantees/f8160290180ba126c904acc32b60eac7/B/2e47e14a986e4838b5afda846a0524da 2023-04-15 02:55:00,780 DEBUG [StoreCloser-TestAcidGuarantees,,1681527274083.f8160290180ba126c904acc32b60eac7.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/f8160290180ba126c904acc32b60eac7/B/0f93ea44f44f4b189b1a30893dbf6367 to hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/archive/data/default/TestAcidGuarantees/f8160290180ba126c904acc32b60eac7/B/0f93ea44f44f4b189b1a30893dbf6367 2023-04-15 02:55:00,781 DEBUG [StoreCloser-TestAcidGuarantees,,1681527274083.f8160290180ba126c904acc32b60eac7.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/f8160290180ba126c904acc32b60eac7/B/0b2852aa58d9444994a5a57c93b0d068 to hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/archive/data/default/TestAcidGuarantees/f8160290180ba126c904acc32b60eac7/B/0b2852aa58d9444994a5a57c93b0d068 2023-04-15 02:55:00,782 DEBUG [StoreCloser-TestAcidGuarantees,,1681527274083.f8160290180ba126c904acc32b60eac7.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/f8160290180ba126c904acc32b60eac7/B/b279733ba1f64aa5af8bfe1a1e7e0cb5 to hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/archive/data/default/TestAcidGuarantees/f8160290180ba126c904acc32b60eac7/B/b279733ba1f64aa5af8bfe1a1e7e0cb5 2023-04-15 02:55:00,782 DEBUG [StoreCloser-TestAcidGuarantees,,1681527274083.f8160290180ba126c904acc32b60eac7.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/f8160290180ba126c904acc32b60eac7/B/ba80e0ee3a2249e79c5e1f0fb611e042 to hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/archive/data/default/TestAcidGuarantees/f8160290180ba126c904acc32b60eac7/B/ba80e0ee3a2249e79c5e1f0fb611e042 2023-04-15 02:55:00,783 DEBUG [StoreCloser-TestAcidGuarantees,,1681527274083.f8160290180ba126c904acc32b60eac7.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/f8160290180ba126c904acc32b60eac7/B/1e7004cf2a474345befc018fc8adcf7f to hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/archive/data/default/TestAcidGuarantees/f8160290180ba126c904acc32b60eac7/B/1e7004cf2a474345befc018fc8adcf7f 2023-04-15 02:55:00,784 DEBUG [StoreCloser-TestAcidGuarantees,,1681527274083.f8160290180ba126c904acc32b60eac7.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/f8160290180ba126c904acc32b60eac7/B/fffc50bcd93a4e7e9a049c6eddc9b678 to hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/archive/data/default/TestAcidGuarantees/f8160290180ba126c904acc32b60eac7/B/fffc50bcd93a4e7e9a049c6eddc9b678 2023-04-15 02:55:00,785 DEBUG [StoreCloser-TestAcidGuarantees,,1681527274083.f8160290180ba126c904acc32b60eac7.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/f8160290180ba126c904acc32b60eac7/B/2b8f9e91423a47e59fbc1a268696347b to hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/archive/data/default/TestAcidGuarantees/f8160290180ba126c904acc32b60eac7/B/2b8f9e91423a47e59fbc1a268696347b 2023-04-15 02:55:00,786 DEBUG [StoreCloser-TestAcidGuarantees,,1681527274083.f8160290180ba126c904acc32b60eac7.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/f8160290180ba126c904acc32b60eac7/B/70be9726a32642b48c6b33242243860b to hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/archive/data/default/TestAcidGuarantees/f8160290180ba126c904acc32b60eac7/B/70be9726a32642b48c6b33242243860b 2023-04-15 02:55:00,786 DEBUG [StoreCloser-TestAcidGuarantees,,1681527274083.f8160290180ba126c904acc32b60eac7.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/f8160290180ba126c904acc32b60eac7/B/e8e29ea90fdc4fdeb8353ff44c9293d2 to hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/archive/data/default/TestAcidGuarantees/f8160290180ba126c904acc32b60eac7/B/e8e29ea90fdc4fdeb8353ff44c9293d2 2023-04-15 02:55:00,787 DEBUG [StoreCloser-TestAcidGuarantees,,1681527274083.f8160290180ba126c904acc32b60eac7.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/f8160290180ba126c904acc32b60eac7/B/f035898cd29a4386814b629068363bef to hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/archive/data/default/TestAcidGuarantees/f8160290180ba126c904acc32b60eac7/B/f035898cd29a4386814b629068363bef 2023-04-15 02:55:00,788 DEBUG [StoreCloser-TestAcidGuarantees,,1681527274083.f8160290180ba126c904acc32b60eac7.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/f8160290180ba126c904acc32b60eac7/B/1af10683cc6a452aaece1b4f85a3c87b to hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/archive/data/default/TestAcidGuarantees/f8160290180ba126c904acc32b60eac7/B/1af10683cc6a452aaece1b4f85a3c87b 2023-04-15 02:55:00,789 DEBUG [StoreCloser-TestAcidGuarantees,,1681527274083.f8160290180ba126c904acc32b60eac7.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/f8160290180ba126c904acc32b60eac7/B/4eca2f71b8374fbd86068b6b3f1136e8 to hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/archive/data/default/TestAcidGuarantees/f8160290180ba126c904acc32b60eac7/B/4eca2f71b8374fbd86068b6b3f1136e8 2023-04-15 02:55:00,790 DEBUG [StoreCloser-TestAcidGuarantees,,1681527274083.f8160290180ba126c904acc32b60eac7.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/f8160290180ba126c904acc32b60eac7/B/9d7fee285c1d4831abc6577d7be84f9a to hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/archive/data/default/TestAcidGuarantees/f8160290180ba126c904acc32b60eac7/B/9d7fee285c1d4831abc6577d7be84f9a 2023-04-15 02:55:00,791 DEBUG [StoreCloser-TestAcidGuarantees,,1681527274083.f8160290180ba126c904acc32b60eac7.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/f8160290180ba126c904acc32b60eac7/B/48c53c106b4642519b29b1e401289ef7 to hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/archive/data/default/TestAcidGuarantees/f8160290180ba126c904acc32b60eac7/B/48c53c106b4642519b29b1e401289ef7 2023-04-15 02:55:00,792 DEBUG [StoreCloser-TestAcidGuarantees,,1681527274083.f8160290180ba126c904acc32b60eac7.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/f8160290180ba126c904acc32b60eac7/B/a87b02c9f45148c6b5788737b0ebb158 to hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/archive/data/default/TestAcidGuarantees/f8160290180ba126c904acc32b60eac7/B/a87b02c9f45148c6b5788737b0ebb158 2023-04-15 02:55:00,793 DEBUG [StoreCloser-TestAcidGuarantees,,1681527274083.f8160290180ba126c904acc32b60eac7.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/f8160290180ba126c904acc32b60eac7/B/cf9b4a5be1174051ab96d62fbd409586 to hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/archive/data/default/TestAcidGuarantees/f8160290180ba126c904acc32b60eac7/B/cf9b4a5be1174051ab96d62fbd409586 2023-04-15 02:55:00,794 DEBUG [StoreCloser-TestAcidGuarantees,,1681527274083.f8160290180ba126c904acc32b60eac7.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/f8160290180ba126c904acc32b60eac7/B/ea1cf09ab444475b96e383a56dc20362 to hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/archive/data/default/TestAcidGuarantees/f8160290180ba126c904acc32b60eac7/B/ea1cf09ab444475b96e383a56dc20362 2023-04-15 02:55:00,795 DEBUG [StoreCloser-TestAcidGuarantees,,1681527274083.f8160290180ba126c904acc32b60eac7.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/f8160290180ba126c904acc32b60eac7/B/a9b63f57015d4c7ba5ccf3f0792317a5 to hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/archive/data/default/TestAcidGuarantees/f8160290180ba126c904acc32b60eac7/B/a9b63f57015d4c7ba5ccf3f0792317a5 2023-04-15 02:55:00,796 DEBUG [StoreCloser-TestAcidGuarantees,,1681527274083.f8160290180ba126c904acc32b60eac7.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/f8160290180ba126c904acc32b60eac7/B/ee48283915b74aa6b528d9fb9f3189f7 to hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/archive/data/default/TestAcidGuarantees/f8160290180ba126c904acc32b60eac7/B/ee48283915b74aa6b528d9fb9f3189f7 2023-04-15 02:55:00,797 DEBUG [StoreCloser-TestAcidGuarantees,,1681527274083.f8160290180ba126c904acc32b60eac7.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/f8160290180ba126c904acc32b60eac7/B/219e1fe89c37434394f36af8af7460ec to hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/archive/data/default/TestAcidGuarantees/f8160290180ba126c904acc32b60eac7/B/219e1fe89c37434394f36af8af7460ec 2023-04-15 02:55:00,798 DEBUG [StoreCloser-TestAcidGuarantees,,1681527274083.f8160290180ba126c904acc32b60eac7.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/f8160290180ba126c904acc32b60eac7/B/f2c8ba28620940748c3055fb37f112be to hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/archive/data/default/TestAcidGuarantees/f8160290180ba126c904acc32b60eac7/B/f2c8ba28620940748c3055fb37f112be 2023-04-15 02:55:00,799 DEBUG [StoreCloser-TestAcidGuarantees,,1681527274083.f8160290180ba126c904acc32b60eac7.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/f8160290180ba126c904acc32b60eac7/B/805958efea2344fdb254357346ad44de to hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/archive/data/default/TestAcidGuarantees/f8160290180ba126c904acc32b60eac7/B/805958efea2344fdb254357346ad44de 2023-04-15 02:55:00,800 DEBUG [StoreCloser-TestAcidGuarantees,,1681527274083.f8160290180ba126c904acc32b60eac7.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/f8160290180ba126c904acc32b60eac7/B/ef70ec718f9246668a85fbea43b0985b to hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/archive/data/default/TestAcidGuarantees/f8160290180ba126c904acc32b60eac7/B/ef70ec718f9246668a85fbea43b0985b 2023-04-15 02:55:00,801 DEBUG [StoreCloser-TestAcidGuarantees,,1681527274083.f8160290180ba126c904acc32b60eac7.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/f8160290180ba126c904acc32b60eac7/B/4b316212595c42cdb3390715265e9446 to hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/archive/data/default/TestAcidGuarantees/f8160290180ba126c904acc32b60eac7/B/4b316212595c42cdb3390715265e9446 2023-04-15 02:55:00,802 DEBUG [StoreCloser-TestAcidGuarantees,,1681527274083.f8160290180ba126c904acc32b60eac7.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/f8160290180ba126c904acc32b60eac7/B/6c158891089b4e7491d4f81163358b6e to hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/archive/data/default/TestAcidGuarantees/f8160290180ba126c904acc32b60eac7/B/6c158891089b4e7491d4f81163358b6e 2023-04-15 02:55:00,805 DEBUG [StoreCloser-TestAcidGuarantees,,1681527274083.f8160290180ba126c904acc32b60eac7.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/f8160290180ba126c904acc32b60eac7/C/925c7d2aa44e4aa1b8865d4572d399ed, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/f8160290180ba126c904acc32b60eac7/C/1c397bc1e228408e8cd10a50d92af44f, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/f8160290180ba126c904acc32b60eac7/C/f4d03398fbf44a50944d5ba548a577d2, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/f8160290180ba126c904acc32b60eac7/C/29656fb2fd6c4e998f291542df8d67fc, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/f8160290180ba126c904acc32b60eac7/C/112f7d90f0894cf7b4f86b1034fb5a3c, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/f8160290180ba126c904acc32b60eac7/C/84dfea7961624d9baeee6e9124300099, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/f8160290180ba126c904acc32b60eac7/C/9cd152c0805f49fdace043f6485e1918, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/f8160290180ba126c904acc32b60eac7/C/3f1b6a9966784dbc8e885ce9ea199958, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/f8160290180ba126c904acc32b60eac7/C/11913411dbc84ae3a331aaf69d76e2ea, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/f8160290180ba126c904acc32b60eac7/C/041c340efc9c4b908be2bec6a35b7f72, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/f8160290180ba126c904acc32b60eac7/C/6ca7e5c7d6c742f49e993eb5d01c790c, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/f8160290180ba126c904acc32b60eac7/C/01c51da9cec44adcb344482e733bcb14, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/f8160290180ba126c904acc32b60eac7/C/3a07d2aaddaa4447a6531f316856f1cc, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/f8160290180ba126c904acc32b60eac7/C/9b2876c82e6d44f0b51ff6f305538d23, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/f8160290180ba126c904acc32b60eac7/C/7807ee8f8b164f238aca4f3363e0ee36, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/f8160290180ba126c904acc32b60eac7/C/378b0b8eb7bd42889792cec86a396fda, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/f8160290180ba126c904acc32b60eac7/C/cecdc7c51df8477f9e6f28d230be6ad0, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/f8160290180ba126c904acc32b60eac7/C/a6113b6a91f3426c92e0b038b548780a, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/f8160290180ba126c904acc32b60eac7/C/40a96079c22f4d509d57c890e94b1486, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/f8160290180ba126c904acc32b60eac7/C/ce7ae2eeb5324a8fbb39fbe803efa5a7, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/f8160290180ba126c904acc32b60eac7/C/0449ab308b884ddf8b41d677a28aff3f, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/f8160290180ba126c904acc32b60eac7/C/6f5b2fbcc01d4e2eb50f02d4ead85ce8, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/f8160290180ba126c904acc32b60eac7/C/f307fe7e6a7e487e87ff809130c5f14e, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/f8160290180ba126c904acc32b60eac7/C/ca8ca18f22304064843cb86197b6fb9f, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/f8160290180ba126c904acc32b60eac7/C/49468c20795140b7bd23c7e29c1bd3e2, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/f8160290180ba126c904acc32b60eac7/C/039dae8ab93444db82666ec43736a590, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/f8160290180ba126c904acc32b60eac7/C/f64cc99821ad4f14926e330d9a735d6b, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/f8160290180ba126c904acc32b60eac7/C/619b1f4e5ede48f18c9511b08d2d11f8, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/f8160290180ba126c904acc32b60eac7/C/ecc6d46705bd4be588a7418471c777ec] to archive 2023-04-15 02:55:00,806 DEBUG [StoreCloser-TestAcidGuarantees,,1681527274083.f8160290180ba126c904acc32b60eac7.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-15 02:55:00,807 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(1227): Checking to see if procedure is done pid=31 2023-04-15 02:55:00,807 DEBUG [StoreCloser-TestAcidGuarantees,,1681527274083.f8160290180ba126c904acc32b60eac7.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/f8160290180ba126c904acc32b60eac7/C/925c7d2aa44e4aa1b8865d4572d399ed to hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/archive/data/default/TestAcidGuarantees/f8160290180ba126c904acc32b60eac7/C/925c7d2aa44e4aa1b8865d4572d399ed 2023-04-15 02:55:00,808 DEBUG [StoreCloser-TestAcidGuarantees,,1681527274083.f8160290180ba126c904acc32b60eac7.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/f8160290180ba126c904acc32b60eac7/C/1c397bc1e228408e8cd10a50d92af44f to hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/archive/data/default/TestAcidGuarantees/f8160290180ba126c904acc32b60eac7/C/1c397bc1e228408e8cd10a50d92af44f 2023-04-15 02:55:00,809 DEBUG [StoreCloser-TestAcidGuarantees,,1681527274083.f8160290180ba126c904acc32b60eac7.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/f8160290180ba126c904acc32b60eac7/C/f4d03398fbf44a50944d5ba548a577d2 to hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/archive/data/default/TestAcidGuarantees/f8160290180ba126c904acc32b60eac7/C/f4d03398fbf44a50944d5ba548a577d2 2023-04-15 02:55:00,810 DEBUG [StoreCloser-TestAcidGuarantees,,1681527274083.f8160290180ba126c904acc32b60eac7.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/f8160290180ba126c904acc32b60eac7/C/29656fb2fd6c4e998f291542df8d67fc to hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/archive/data/default/TestAcidGuarantees/f8160290180ba126c904acc32b60eac7/C/29656fb2fd6c4e998f291542df8d67fc 2023-04-15 02:55:00,811 DEBUG [StoreCloser-TestAcidGuarantees,,1681527274083.f8160290180ba126c904acc32b60eac7.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/f8160290180ba126c904acc32b60eac7/C/112f7d90f0894cf7b4f86b1034fb5a3c to hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/archive/data/default/TestAcidGuarantees/f8160290180ba126c904acc32b60eac7/C/112f7d90f0894cf7b4f86b1034fb5a3c 2023-04-15 02:55:00,812 DEBUG [StoreCloser-TestAcidGuarantees,,1681527274083.f8160290180ba126c904acc32b60eac7.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/f8160290180ba126c904acc32b60eac7/C/84dfea7961624d9baeee6e9124300099 to hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/archive/data/default/TestAcidGuarantees/f8160290180ba126c904acc32b60eac7/C/84dfea7961624d9baeee6e9124300099 2023-04-15 02:55:00,812 DEBUG [StoreCloser-TestAcidGuarantees,,1681527274083.f8160290180ba126c904acc32b60eac7.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/f8160290180ba126c904acc32b60eac7/C/9cd152c0805f49fdace043f6485e1918 to hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/archive/data/default/TestAcidGuarantees/f8160290180ba126c904acc32b60eac7/C/9cd152c0805f49fdace043f6485e1918 2023-04-15 02:55:00,813 DEBUG [StoreCloser-TestAcidGuarantees,,1681527274083.f8160290180ba126c904acc32b60eac7.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/f8160290180ba126c904acc32b60eac7/C/3f1b6a9966784dbc8e885ce9ea199958 to hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/archive/data/default/TestAcidGuarantees/f8160290180ba126c904acc32b60eac7/C/3f1b6a9966784dbc8e885ce9ea199958 2023-04-15 02:55:00,814 DEBUG [StoreCloser-TestAcidGuarantees,,1681527274083.f8160290180ba126c904acc32b60eac7.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/f8160290180ba126c904acc32b60eac7/C/11913411dbc84ae3a331aaf69d76e2ea to hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/archive/data/default/TestAcidGuarantees/f8160290180ba126c904acc32b60eac7/C/11913411dbc84ae3a331aaf69d76e2ea 2023-04-15 02:55:00,815 DEBUG [StoreCloser-TestAcidGuarantees,,1681527274083.f8160290180ba126c904acc32b60eac7.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/f8160290180ba126c904acc32b60eac7/C/041c340efc9c4b908be2bec6a35b7f72 to hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/archive/data/default/TestAcidGuarantees/f8160290180ba126c904acc32b60eac7/C/041c340efc9c4b908be2bec6a35b7f72 2023-04-15 02:55:00,816 DEBUG [StoreCloser-TestAcidGuarantees,,1681527274083.f8160290180ba126c904acc32b60eac7.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/f8160290180ba126c904acc32b60eac7/C/6ca7e5c7d6c742f49e993eb5d01c790c to hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/archive/data/default/TestAcidGuarantees/f8160290180ba126c904acc32b60eac7/C/6ca7e5c7d6c742f49e993eb5d01c790c 2023-04-15 02:55:00,816 DEBUG [StoreCloser-TestAcidGuarantees,,1681527274083.f8160290180ba126c904acc32b60eac7.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/f8160290180ba126c904acc32b60eac7/C/01c51da9cec44adcb344482e733bcb14 to hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/archive/data/default/TestAcidGuarantees/f8160290180ba126c904acc32b60eac7/C/01c51da9cec44adcb344482e733bcb14 2023-04-15 02:55:00,817 DEBUG [StoreCloser-TestAcidGuarantees,,1681527274083.f8160290180ba126c904acc32b60eac7.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/f8160290180ba126c904acc32b60eac7/C/3a07d2aaddaa4447a6531f316856f1cc to hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/archive/data/default/TestAcidGuarantees/f8160290180ba126c904acc32b60eac7/C/3a07d2aaddaa4447a6531f316856f1cc 2023-04-15 02:55:00,818 DEBUG [StoreCloser-TestAcidGuarantees,,1681527274083.f8160290180ba126c904acc32b60eac7.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/f8160290180ba126c904acc32b60eac7/C/9b2876c82e6d44f0b51ff6f305538d23 to hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/archive/data/default/TestAcidGuarantees/f8160290180ba126c904acc32b60eac7/C/9b2876c82e6d44f0b51ff6f305538d23 2023-04-15 02:55:00,819 DEBUG [StoreCloser-TestAcidGuarantees,,1681527274083.f8160290180ba126c904acc32b60eac7.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/f8160290180ba126c904acc32b60eac7/C/7807ee8f8b164f238aca4f3363e0ee36 to hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/archive/data/default/TestAcidGuarantees/f8160290180ba126c904acc32b60eac7/C/7807ee8f8b164f238aca4f3363e0ee36 2023-04-15 02:55:00,820 DEBUG [StoreCloser-TestAcidGuarantees,,1681527274083.f8160290180ba126c904acc32b60eac7.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/f8160290180ba126c904acc32b60eac7/C/378b0b8eb7bd42889792cec86a396fda to hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/archive/data/default/TestAcidGuarantees/f8160290180ba126c904acc32b60eac7/C/378b0b8eb7bd42889792cec86a396fda 2023-04-15 02:55:00,821 DEBUG [StoreCloser-TestAcidGuarantees,,1681527274083.f8160290180ba126c904acc32b60eac7.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/f8160290180ba126c904acc32b60eac7/C/cecdc7c51df8477f9e6f28d230be6ad0 to hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/archive/data/default/TestAcidGuarantees/f8160290180ba126c904acc32b60eac7/C/cecdc7c51df8477f9e6f28d230be6ad0 2023-04-15 02:55:00,822 DEBUG [StoreCloser-TestAcidGuarantees,,1681527274083.f8160290180ba126c904acc32b60eac7.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/f8160290180ba126c904acc32b60eac7/C/a6113b6a91f3426c92e0b038b548780a to hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/archive/data/default/TestAcidGuarantees/f8160290180ba126c904acc32b60eac7/C/a6113b6a91f3426c92e0b038b548780a 2023-04-15 02:55:00,823 DEBUG [StoreCloser-TestAcidGuarantees,,1681527274083.f8160290180ba126c904acc32b60eac7.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/f8160290180ba126c904acc32b60eac7/C/40a96079c22f4d509d57c890e94b1486 to hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/archive/data/default/TestAcidGuarantees/f8160290180ba126c904acc32b60eac7/C/40a96079c22f4d509d57c890e94b1486 2023-04-15 02:55:00,824 DEBUG [StoreCloser-TestAcidGuarantees,,1681527274083.f8160290180ba126c904acc32b60eac7.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/f8160290180ba126c904acc32b60eac7/C/ce7ae2eeb5324a8fbb39fbe803efa5a7 to hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/archive/data/default/TestAcidGuarantees/f8160290180ba126c904acc32b60eac7/C/ce7ae2eeb5324a8fbb39fbe803efa5a7 2023-04-15 02:55:00,824 DEBUG [StoreCloser-TestAcidGuarantees,,1681527274083.f8160290180ba126c904acc32b60eac7.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/f8160290180ba126c904acc32b60eac7/C/0449ab308b884ddf8b41d677a28aff3f to hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/archive/data/default/TestAcidGuarantees/f8160290180ba126c904acc32b60eac7/C/0449ab308b884ddf8b41d677a28aff3f 2023-04-15 02:55:00,825 DEBUG [StoreCloser-TestAcidGuarantees,,1681527274083.f8160290180ba126c904acc32b60eac7.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/f8160290180ba126c904acc32b60eac7/C/6f5b2fbcc01d4e2eb50f02d4ead85ce8 to hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/archive/data/default/TestAcidGuarantees/f8160290180ba126c904acc32b60eac7/C/6f5b2fbcc01d4e2eb50f02d4ead85ce8 2023-04-15 02:55:00,826 DEBUG [StoreCloser-TestAcidGuarantees,,1681527274083.f8160290180ba126c904acc32b60eac7.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/f8160290180ba126c904acc32b60eac7/C/f307fe7e6a7e487e87ff809130c5f14e to hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/archive/data/default/TestAcidGuarantees/f8160290180ba126c904acc32b60eac7/C/f307fe7e6a7e487e87ff809130c5f14e 2023-04-15 02:55:00,827 DEBUG [StoreCloser-TestAcidGuarantees,,1681527274083.f8160290180ba126c904acc32b60eac7.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/f8160290180ba126c904acc32b60eac7/C/ca8ca18f22304064843cb86197b6fb9f to hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/archive/data/default/TestAcidGuarantees/f8160290180ba126c904acc32b60eac7/C/ca8ca18f22304064843cb86197b6fb9f 2023-04-15 02:55:00,828 DEBUG [StoreCloser-TestAcidGuarantees,,1681527274083.f8160290180ba126c904acc32b60eac7.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/f8160290180ba126c904acc32b60eac7/C/49468c20795140b7bd23c7e29c1bd3e2 to hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/archive/data/default/TestAcidGuarantees/f8160290180ba126c904acc32b60eac7/C/49468c20795140b7bd23c7e29c1bd3e2 2023-04-15 02:55:00,829 DEBUG [StoreCloser-TestAcidGuarantees,,1681527274083.f8160290180ba126c904acc32b60eac7.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/f8160290180ba126c904acc32b60eac7/C/039dae8ab93444db82666ec43736a590 to hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/archive/data/default/TestAcidGuarantees/f8160290180ba126c904acc32b60eac7/C/039dae8ab93444db82666ec43736a590 2023-04-15 02:55:00,830 DEBUG [StoreCloser-TestAcidGuarantees,,1681527274083.f8160290180ba126c904acc32b60eac7.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/f8160290180ba126c904acc32b60eac7/C/f64cc99821ad4f14926e330d9a735d6b to hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/archive/data/default/TestAcidGuarantees/f8160290180ba126c904acc32b60eac7/C/f64cc99821ad4f14926e330d9a735d6b 2023-04-15 02:55:00,831 DEBUG [StoreCloser-TestAcidGuarantees,,1681527274083.f8160290180ba126c904acc32b60eac7.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/f8160290180ba126c904acc32b60eac7/C/619b1f4e5ede48f18c9511b08d2d11f8 to hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/archive/data/default/TestAcidGuarantees/f8160290180ba126c904acc32b60eac7/C/619b1f4e5ede48f18c9511b08d2d11f8 2023-04-15 02:55:00,832 DEBUG [StoreCloser-TestAcidGuarantees,,1681527274083.f8160290180ba126c904acc32b60eac7.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/f8160290180ba126c904acc32b60eac7/C/ecc6d46705bd4be588a7418471c777ec to hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/archive/data/default/TestAcidGuarantees/f8160290180ba126c904acc32b60eac7/C/ecc6d46705bd4be588a7418471c777ec 2023-04-15 02:55:00,838 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/f8160290180ba126c904acc32b60eac7/recovered.edits/616.seqid, newMaxSeqId=616, maxSeqId=1 2023-04-15 02:55:00,839 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1681527274083.f8160290180ba126c904acc32b60eac7. 2023-04-15 02:55:00,839 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1558): Region close journal for f8160290180ba126c904acc32b60eac7: 2023-04-15 02:55:00,841 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] handler.UnassignRegionHandler(147): Closed f8160290180ba126c904acc32b60eac7 2023-04-15 02:55:00,841 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=32 updating hbase:meta row=f8160290180ba126c904acc32b60eac7, regionState=CLOSED 2023-04-15 02:55:00,841 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1681527274083.f8160290180ba126c904acc32b60eac7.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1681527300841"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1681527300841"}]},"ts":"1681527300841"} 2023-04-15 02:55:00,843 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=33, resume processing ppid=32 2023-04-15 02:55:00,843 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=33, ppid=32, state=SUCCESS; CloseRegionProcedure f8160290180ba126c904acc32b60eac7, server=jenkins-hbase20.apache.org,34681,1681527209133 in 1.1330 sec 2023-04-15 02:55:00,844 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=32, resume processing ppid=31 2023-04-15 02:55:00,844 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=32, ppid=31, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=f8160290180ba126c904acc32b60eac7, UNASSIGN in 1.1370 sec 2023-04-15 02:55:00,844 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1681527300844"}]},"ts":"1681527300844"} 2023-04-15 02:55:00,845 INFO [PEWorker-5] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=DISABLED in hbase:meta 2023-04-15 02:55:00,846 INFO [PEWorker-5] procedure.DisableTableProcedure(305): Set TestAcidGuarantees to state=DISABLED 2023-04-15 02:55:00,847 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=31, state=SUCCESS; DisableTableProcedure table=TestAcidGuarantees in 1.1460 sec 2023-04-15 02:55:01,808 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(1227): Checking to see if procedure is done pid=31 2023-04-15 02:55:01,808 INFO [Listener at localhost.localdomain/33693] client.HBaseAdmin$TableFuture(3541): Operation: DISABLE, Table Name: default:TestAcidGuarantees, procId: 31 completed 2023-04-15 02:55:01,809 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.HMaster$5(2228): Client=jenkins//148.251.75.209 delete TestAcidGuarantees 2023-04-15 02:55:01,810 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] procedure2.ProcedureExecutor(1029): Stored pid=34, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION; DeleteTableProcedure table=TestAcidGuarantees 2023-04-15 02:55:01,811 DEBUG [PEWorker-4] procedure.DeleteTableProcedure(101): Waiting for RIT for pid=34, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-15 02:55:01,812 DEBUG [PEWorker-4] procedure.DeleteTableProcedure(113): Deleting regions from filesystem for pid=34, state=RUNNABLE:DELETE_TABLE_CLEAR_FS_LAYOUT, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-15 02:55:01,812 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(1227): Checking to see if procedure is done pid=34 2023-04-15 02:55:01,816 DEBUG [HFileArchiver-7] backup.HFileArchiver(131): ARCHIVING hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/.tmp/data/default/TestAcidGuarantees/f8160290180ba126c904acc32b60eac7 2023-04-15 02:55:01,820 DEBUG [HFileArchiver-7] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/.tmp/data/default/TestAcidGuarantees/f8160290180ba126c904acc32b60eac7/A, FileablePath, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/.tmp/data/default/TestAcidGuarantees/f8160290180ba126c904acc32b60eac7/B, FileablePath, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/.tmp/data/default/TestAcidGuarantees/f8160290180ba126c904acc32b60eac7/C, FileablePath, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/.tmp/data/default/TestAcidGuarantees/f8160290180ba126c904acc32b60eac7/recovered.edits] 2023-04-15 02:55:01,823 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/.tmp/data/default/TestAcidGuarantees/f8160290180ba126c904acc32b60eac7/A/371895e17d3c491481f5336a21a159e6 to hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/archive/data/default/TestAcidGuarantees/f8160290180ba126c904acc32b60eac7/A/371895e17d3c491481f5336a21a159e6 2023-04-15 02:55:01,832 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/.tmp/data/default/TestAcidGuarantees/f8160290180ba126c904acc32b60eac7/A/818405efe3054b6c80231b64e1ed28b1 to hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/archive/data/default/TestAcidGuarantees/f8160290180ba126c904acc32b60eac7/A/818405efe3054b6c80231b64e1ed28b1 2023-04-15 02:55:01,833 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/.tmp/data/default/TestAcidGuarantees/f8160290180ba126c904acc32b60eac7/A/e41d6a86a022461ca5b1042b208c8158 to hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/archive/data/default/TestAcidGuarantees/f8160290180ba126c904acc32b60eac7/A/e41d6a86a022461ca5b1042b208c8158 2023-04-15 02:55:01,836 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/.tmp/data/default/TestAcidGuarantees/f8160290180ba126c904acc32b60eac7/B/3d5089a21ac3492b8ab691a71b9f7ba3 to hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/archive/data/default/TestAcidGuarantees/f8160290180ba126c904acc32b60eac7/B/3d5089a21ac3492b8ab691a71b9f7ba3 2023-04-15 02:55:01,837 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/.tmp/data/default/TestAcidGuarantees/f8160290180ba126c904acc32b60eac7/B/43de67ec7bb1430ba0ea05932a731fd1 to hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/archive/data/default/TestAcidGuarantees/f8160290180ba126c904acc32b60eac7/B/43de67ec7bb1430ba0ea05932a731fd1 2023-04-15 02:55:01,838 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/.tmp/data/default/TestAcidGuarantees/f8160290180ba126c904acc32b60eac7/B/ed1b2246733d494382852c282b1727b0 to hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/archive/data/default/TestAcidGuarantees/f8160290180ba126c904acc32b60eac7/B/ed1b2246733d494382852c282b1727b0 2023-04-15 02:55:01,843 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/.tmp/data/default/TestAcidGuarantees/f8160290180ba126c904acc32b60eac7/C/20f1b0d0f922450b8088c359da4dc910 to hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/archive/data/default/TestAcidGuarantees/f8160290180ba126c904acc32b60eac7/C/20f1b0d0f922450b8088c359da4dc910 2023-04-15 02:55:01,844 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/.tmp/data/default/TestAcidGuarantees/f8160290180ba126c904acc32b60eac7/C/9cc512d9db0d48789e4ecb33dfb397fe to hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/archive/data/default/TestAcidGuarantees/f8160290180ba126c904acc32b60eac7/C/9cc512d9db0d48789e4ecb33dfb397fe 2023-04-15 02:55:01,846 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/.tmp/data/default/TestAcidGuarantees/f8160290180ba126c904acc32b60eac7/C/ccb42311a7bb487cad982f7836f0e907 to hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/archive/data/default/TestAcidGuarantees/f8160290180ba126c904acc32b60eac7/C/ccb42311a7bb487cad982f7836f0e907 2023-04-15 02:55:01,848 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/.tmp/data/default/TestAcidGuarantees/f8160290180ba126c904acc32b60eac7/recovered.edits/616.seqid to hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/archive/data/default/TestAcidGuarantees/f8160290180ba126c904acc32b60eac7/recovered.edits/616.seqid 2023-04-15 02:55:01,849 DEBUG [HFileArchiver-7] backup.HFileArchiver(596): Deleted hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/.tmp/data/default/TestAcidGuarantees/f8160290180ba126c904acc32b60eac7 2023-04-15 02:55:01,849 DEBUG [PEWorker-4] procedure.DeleteTableProcedure(328): Archived TestAcidGuarantees regions 2023-04-15 02:55:01,851 DEBUG [PEWorker-4] procedure.DeleteTableProcedure(118): Deleting regions from META for pid=34, state=RUNNABLE:DELETE_TABLE_REMOVE_FROM_META, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-15 02:55:01,853 WARN [PEWorker-4] procedure.DeleteTableProcedure(384): Deleting some vestigial 1 rows of TestAcidGuarantees from hbase:meta 2023-04-15 02:55:01,855 DEBUG [PEWorker-4] procedure.DeleteTableProcedure(421): Removing 'TestAcidGuarantees' descriptor. 2023-04-15 02:55:01,856 DEBUG [PEWorker-4] procedure.DeleteTableProcedure(124): Deleting assignment state for pid=34, state=RUNNABLE:DELETE_TABLE_UNASSIGN_REGIONS, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-15 02:55:01,856 DEBUG [PEWorker-4] procedure.DeleteTableProcedure(411): Removing 'TestAcidGuarantees' from region states. 2023-04-15 02:55:01,856 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"TestAcidGuarantees,,1681527274083.f8160290180ba126c904acc32b60eac7.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1681527301856"}]},"ts":"9223372036854775807"} 2023-04-15 02:55:01,859 INFO [PEWorker-4] hbase.MetaTableAccessor(1788): Deleted 1 regions from META 2023-04-15 02:55:01,859 DEBUG [PEWorker-4] hbase.MetaTableAccessor(1789): Deleted regions: [{ENCODED => f8160290180ba126c904acc32b60eac7, NAME => 'TestAcidGuarantees,,1681527274083.f8160290180ba126c904acc32b60eac7.', STARTKEY => '', ENDKEY => ''}] 2023-04-15 02:55:01,860 DEBUG [PEWorker-4] procedure.DeleteTableProcedure(415): Marking 'TestAcidGuarantees' as deleted. 2023-04-15 02:55:01,860 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":0,"tag":[],"timestamp":"1681527301860"}]},"ts":"9223372036854775807"} 2023-04-15 02:55:01,861 INFO [PEWorker-4] hbase.MetaTableAccessor(1658): Deleted table TestAcidGuarantees state from META 2023-04-15 02:55:01,864 DEBUG [PEWorker-4] procedure.DeleteTableProcedure(130): Finished pid=34, state=RUNNABLE:DELETE_TABLE_POST_OPERATION, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-15 02:55:01,865 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=34, state=SUCCESS; DeleteTableProcedure table=TestAcidGuarantees in 55 msec 2023-04-15 02:55:01,913 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(1227): Checking to see if procedure is done pid=34 2023-04-15 02:55:01,913 INFO [Listener at localhost.localdomain/33693] client.HBaseAdmin$TableFuture(3541): Operation: DELETE, Table Name: default:TestAcidGuarantees, procId: 34 completed 2023-04-15 02:55:01,927 INFO [Listener at localhost.localdomain/33693] hbase.ResourceChecker(175): after: TestAcidGuaranteesWithEagerPolicy#testGetAtomicity Thread=226 (was 224) - Thread LEAK? -, OpenFileDescriptor=553 (was 517) - OpenFileDescriptor LEAK? -, MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=380 (was 373) - SystemLoadAverage LEAK? -, ProcessCount=169 (was 169), AvailableMemoryMB=5736 (was 5720) - AvailableMemoryMB LEAK? - 2023-04-15 02:55:01,939 INFO [Listener at localhost.localdomain/33693] hbase.ResourceChecker(147): before: TestAcidGuaranteesWithEagerPolicy#testMobScanAtomicity Thread=226, OpenFileDescriptor=553, MaxFileDescriptor=60000, SystemLoadAverage=380, ProcessCount=169, AvailableMemoryMB=5735 2023-04-15 02:55:01,941 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] util.TableDescriptorChecker(340): MEMSTORE_FLUSHSIZE for table descriptor or "hbase.hregion.memstore.flush.size" (131072) is too small, which might cause very frequent flushing. 2023-04-15 02:55:01,941 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.HMaster$4(2112): Client=jenkins//148.251.75.209 create 'TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-04-15 02:55:01,942 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] procedure2.ProcedureExecutor(1029): Stored pid=35, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION; CreateTableProcedure table=TestAcidGuarantees 2023-04-15 02:55:01,943 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=35, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_PRE_OPERATION 2023-04-15 02:55:01,943 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(697): Client=jenkins//148.251.75.209 procedure request for creating table: namespace: "default" qualifier: "TestAcidGuarantees" procId is: 35 2023-04-15 02:55:01,944 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=35, state=RUNNABLE:CREATE_TABLE_WRITE_FS_LAYOUT, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_WRITE_FS_LAYOUT 2023-04-15 02:55:01,944 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(1227): Checking to see if procedure is done pid=35 2023-04-15 02:55:01,945 DEBUG [HFileArchiver-8] backup.HFileArchiver(131): ARCHIVING hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/.tmp/data/default/TestAcidGuarantees/104a1f3a98692b3503aef02f1d865942 2023-04-15 02:55:01,945 DEBUG [HFileArchiver-8] backup.HFileArchiver(153): Directory hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/.tmp/data/default/TestAcidGuarantees/104a1f3a98692b3503aef02f1d865942 empty. 2023-04-15 02:55:01,946 DEBUG [HFileArchiver-8] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/.tmp/data/default/TestAcidGuarantees/104a1f3a98692b3503aef02f1d865942 2023-04-15 02:55:01,946 DEBUG [PEWorker-3] procedure.DeleteTableProcedure(328): Archived TestAcidGuarantees regions 2023-04-15 02:55:02,045 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(1227): Checking to see if procedure is done pid=35 2023-04-15 02:55:02,246 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(1227): Checking to see if procedure is done pid=35 2023-04-15 02:55:02,353 DEBUG [PEWorker-3] util.FSTableDescriptors(570): Wrote into hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/.tmp/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000001 2023-04-15 02:55:02,354 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(7675): creating {ENCODED => 104a1f3a98692b3503aef02f1d865942, NAME => 'TestAcidGuarantees,,1681527301940.104a1f3a98692b3503aef02f1d865942.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/.tmp 2023-04-15 02:55:02,547 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(1227): Checking to see if procedure is done pid=35 2023-04-15 02:55:02,760 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1681527301940.104a1f3a98692b3503aef02f1d865942.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-15 02:55:02,760 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1604): Closing 104a1f3a98692b3503aef02f1d865942, disabling compactions & flushes 2023-04-15 02:55:02,760 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1681527301940.104a1f3a98692b3503aef02f1d865942. 2023-04-15 02:55:02,760 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1681527301940.104a1f3a98692b3503aef02f1d865942. 2023-04-15 02:55:02,760 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1681527301940.104a1f3a98692b3503aef02f1d865942. after waiting 0 ms 2023-04-15 02:55:02,760 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1681527301940.104a1f3a98692b3503aef02f1d865942. 2023-04-15 02:55:02,760 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1681527301940.104a1f3a98692b3503aef02f1d865942. 2023-04-15 02:55:02,760 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1558): Region close journal for 104a1f3a98692b3503aef02f1d865942: 2023-04-15 02:55:02,762 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=35, state=RUNNABLE:CREATE_TABLE_ADD_TO_META, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_ADD_TO_META 2023-04-15 02:55:02,763 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1681527301940.104a1f3a98692b3503aef02f1d865942.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1681527302763"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1681527302763"}]},"ts":"1681527302763"} 2023-04-15 02:55:02,764 INFO [PEWorker-3] hbase.MetaTableAccessor(1496): Added 1 regions to meta. 2023-04-15 02:55:02,764 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=35, state=RUNNABLE:CREATE_TABLE_ASSIGN_REGIONS, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_ASSIGN_REGIONS 2023-04-15 02:55:02,764 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1681527302764"}]},"ts":"1681527302764"} 2023-04-15 02:55:02,765 INFO [PEWorker-3] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=ENABLING in hbase:meta 2023-04-15 02:55:02,767 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=36, ppid=35, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=TestAcidGuarantees, region=104a1f3a98692b3503aef02f1d865942, ASSIGN}] 2023-04-15 02:55:02,769 INFO [PEWorker-2] procedure.MasterProcedureScheduler(727): Took xlock for pid=36, ppid=35, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=TestAcidGuarantees, region=104a1f3a98692b3503aef02f1d865942, ASSIGN 2023-04-15 02:55:02,770 INFO [PEWorker-2] assignment.TransitRegionStateProcedure(193): Starting pid=36, ppid=35, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=TestAcidGuarantees, region=104a1f3a98692b3503aef02f1d865942, ASSIGN; state=OFFLINE, location=jenkins-hbase20.apache.org,34681,1681527209133; forceNewPlan=false, retain=false 2023-04-15 02:55:02,921 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=36 updating hbase:meta row=104a1f3a98692b3503aef02f1d865942, regionState=OPENING, regionLocation=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:02,922 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1681527301940.104a1f3a98692b3503aef02f1d865942.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1681527302921"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1681527302921"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1681527302921"}]},"ts":"1681527302921"} 2023-04-15 02:55:02,923 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=37, ppid=36, state=RUNNABLE; OpenRegionProcedure 104a1f3a98692b3503aef02f1d865942, server=jenkins-hbase20.apache.org,34681,1681527209133}] 2023-04-15 02:55:03,048 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(1227): Checking to see if procedure is done pid=35 2023-04-15 02:55:03,078 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] handler.AssignRegionHandler(128): Open TestAcidGuarantees,,1681527301940.104a1f3a98692b3503aef02f1d865942. 2023-04-15 02:55:03,078 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => 104a1f3a98692b3503aef02f1d865942, NAME => 'TestAcidGuarantees,,1681527301940.104a1f3a98692b3503aef02f1d865942.', STARTKEY => '', ENDKEY => ''} 2023-04-15 02:55:03,078 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table TestAcidGuarantees 104a1f3a98692b3503aef02f1d865942 2023-04-15 02:55:03,078 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1681527301940.104a1f3a98692b3503aef02f1d865942.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-15 02:55:03,078 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7894): checking encryption for 104a1f3a98692b3503aef02f1d865942 2023-04-15 02:55:03,078 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7897): checking classloading for 104a1f3a98692b3503aef02f1d865942 2023-04-15 02:55:03,079 INFO [StoreOpener-104a1f3a98692b3503aef02f1d865942-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family A of region 104a1f3a98692b3503aef02f1d865942 2023-04-15 02:55:03,081 DEBUG [StoreOpener-104a1f3a98692b3503aef02f1d865942-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/104a1f3a98692b3503aef02f1d865942/A 2023-04-15 02:55:03,081 DEBUG [StoreOpener-104a1f3a98692b3503aef02f1d865942-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/104a1f3a98692b3503aef02f1d865942/A 2023-04-15 02:55:03,081 INFO [StoreOpener-104a1f3a98692b3503aef02f1d865942-1] regionserver.CompactingMemStore(121): Store=A, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-15 02:55:03,081 INFO [StoreOpener-104a1f3a98692b3503aef02f1d865942-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 104a1f3a98692b3503aef02f1d865942 columnFamilyName A 2023-04-15 02:55:03,082 INFO [StoreOpener-104a1f3a98692b3503aef02f1d865942-1] regionserver.HStore(310): Store=104a1f3a98692b3503aef02f1d865942/A, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-15 02:55:03,082 INFO [StoreOpener-104a1f3a98692b3503aef02f1d865942-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family B of region 104a1f3a98692b3503aef02f1d865942 2023-04-15 02:55:03,083 DEBUG [StoreOpener-104a1f3a98692b3503aef02f1d865942-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/104a1f3a98692b3503aef02f1d865942/B 2023-04-15 02:55:03,083 DEBUG [StoreOpener-104a1f3a98692b3503aef02f1d865942-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/104a1f3a98692b3503aef02f1d865942/B 2023-04-15 02:55:03,083 INFO [StoreOpener-104a1f3a98692b3503aef02f1d865942-1] regionserver.CompactingMemStore(121): Store=B, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-15 02:55:03,083 INFO [StoreOpener-104a1f3a98692b3503aef02f1d865942-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 104a1f3a98692b3503aef02f1d865942 columnFamilyName B 2023-04-15 02:55:03,084 INFO [StoreOpener-104a1f3a98692b3503aef02f1d865942-1] regionserver.HStore(310): Store=104a1f3a98692b3503aef02f1d865942/B, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-15 02:55:03,084 INFO [StoreOpener-104a1f3a98692b3503aef02f1d865942-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family C of region 104a1f3a98692b3503aef02f1d865942 2023-04-15 02:55:03,085 DEBUG [StoreOpener-104a1f3a98692b3503aef02f1d865942-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/104a1f3a98692b3503aef02f1d865942/C 2023-04-15 02:55:03,085 DEBUG [StoreOpener-104a1f3a98692b3503aef02f1d865942-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/104a1f3a98692b3503aef02f1d865942/C 2023-04-15 02:55:03,085 INFO [StoreOpener-104a1f3a98692b3503aef02f1d865942-1] regionserver.CompactingMemStore(121): Store=C, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-15 02:55:03,085 INFO [StoreOpener-104a1f3a98692b3503aef02f1d865942-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 104a1f3a98692b3503aef02f1d865942 columnFamilyName C 2023-04-15 02:55:03,086 INFO [StoreOpener-104a1f3a98692b3503aef02f1d865942-1] regionserver.HStore(310): Store=104a1f3a98692b3503aef02f1d865942/C, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-15 02:55:03,086 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1146): Setting FlushNonSloppyStoresFirstPolicy for the region=TestAcidGuarantees,,1681527301940.104a1f3a98692b3503aef02f1d865942. 2023-04-15 02:55:03,087 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/104a1f3a98692b3503aef02f1d865942 2023-04-15 02:55:03,087 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/104a1f3a98692b3503aef02f1d865942 2023-04-15 02:55:03,089 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table TestAcidGuarantees descriptor;using region.getMemStoreFlushHeapSize/# of families (16.0 M)) instead. 2023-04-15 02:55:03,090 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1055): writing seq id for 104a1f3a98692b3503aef02f1d865942 2023-04-15 02:55:03,092 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/104a1f3a98692b3503aef02f1d865942/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-04-15 02:55:03,092 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1072): Opened 104a1f3a98692b3503aef02f1d865942; next sequenceid=2; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=67828069, jitterRate=0.010716989636421204}, FlushLargeStoresPolicy{flushSizeLowerBound=16777216} 2023-04-15 02:55:03,092 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(965): Region open journal for 104a1f3a98692b3503aef02f1d865942: 2023-04-15 02:55:03,093 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for TestAcidGuarantees,,1681527301940.104a1f3a98692b3503aef02f1d865942., pid=37, masterSystemTime=1681527303075 2023-04-15 02:55:03,094 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for TestAcidGuarantees,,1681527301940.104a1f3a98692b3503aef02f1d865942. 2023-04-15 02:55:03,094 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] handler.AssignRegionHandler(156): Opened TestAcidGuarantees,,1681527301940.104a1f3a98692b3503aef02f1d865942. 2023-04-15 02:55:03,094 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=36 updating hbase:meta row=104a1f3a98692b3503aef02f1d865942, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:03,095 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"TestAcidGuarantees,,1681527301940.104a1f3a98692b3503aef02f1d865942.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1681527303094"},{"qualifier":"server","vlen":32,"tag":[],"timestamp":"1681527303094"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1681527303094"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1681527303094"}]},"ts":"1681527303094"} 2023-04-15 02:55:03,097 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=37, resume processing ppid=36 2023-04-15 02:55:03,097 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=37, ppid=36, state=SUCCESS; OpenRegionProcedure 104a1f3a98692b3503aef02f1d865942, server=jenkins-hbase20.apache.org,34681,1681527209133 in 172 msec 2023-04-15 02:55:03,099 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=36, resume processing ppid=35 2023-04-15 02:55:03,099 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=36, ppid=35, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=104a1f3a98692b3503aef02f1d865942, ASSIGN in 330 msec 2023-04-15 02:55:03,099 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=35, state=RUNNABLE:CREATE_TABLE_UPDATE_DESC_CACHE, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_UPDATE_DESC_CACHE 2023-04-15 02:55:03,100 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1681527303099"}]},"ts":"1681527303099"} 2023-04-15 02:55:03,100 INFO [PEWorker-2] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=ENABLED in hbase:meta 2023-04-15 02:55:03,102 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=35, state=RUNNABLE:CREATE_TABLE_POST_OPERATION, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_POST_OPERATION 2023-04-15 02:55:03,103 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=35, state=SUCCESS; CreateTableProcedure table=TestAcidGuarantees in 1.1610 sec 2023-04-15 02:55:04,049 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(1227): Checking to see if procedure is done pid=35 2023-04-15 02:55:04,050 INFO [Listener at localhost.localdomain/33693] client.HBaseAdmin$TableFuture(3541): Operation: CREATE, Table Name: default:TestAcidGuarantees, procId: 35 completed 2023-04-15 02:55:04,051 DEBUG [Listener at localhost.localdomain/33693] zookeeper.ReadOnlyZKClient(139): Connect 0x7a0240a4 to 127.0.0.1:51762 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-15 02:55:04,058 DEBUG [Listener at localhost.localdomain/33693] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@59e327cd, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-15 02:55:04,059 DEBUG [Listener at localhost.localdomain/33693] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-15 02:55:04,061 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:41110, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-15 02:55:04,063 DEBUG [Listener at localhost.localdomain/33693] ipc.RpcConnection(124): Using SIMPLE authentication for service=MasterService, sasl=false 2023-04-15 02:55:04,064 INFO [RS-EventLoopGroup-1-1] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:40504, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=MasterService 2023-04-15 02:55:04,066 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] util.TableDescriptorChecker(340): MEMSTORE_FLUSHSIZE for table descriptor or "hbase.hregion.memstore.flush.size" (131072) is too small, which might cause very frequent flushing. 2023-04-15 02:55:04,066 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.HMaster$12(2454): Client=jenkins//148.251.75.209 modify table TestAcidGuarantees from 'TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} to 'TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', MOB_THRESHOLD => '4', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-04-15 02:55:04,068 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] procedure2.ProcedureExecutor(1029): Stored pid=38, state=RUNNABLE:MODIFY_TABLE_PREPARE; ModifyTableProcedure table=TestAcidGuarantees 2023-04-15 02:55:04,485 DEBUG [PEWorker-1] util.FSTableDescriptors(570): Wrote into hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000002 2023-04-15 02:55:04,487 DEBUG [PEWorker-1] util.FSTableDescriptors(519): Deleted hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000001 2023-04-15 02:55:04,487 INFO [PEWorker-1] util.FSTableDescriptors(293): Updated tableinfo=hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000002 2023-04-15 02:55:04,491 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=39, ppid=38, state=RUNNABLE:REOPEN_TABLE_REGIONS_GET_REGIONS; ReopenTableRegionsProcedure table=TestAcidGuarantees}] 2023-04-15 02:55:04,495 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=40, ppid=39, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=104a1f3a98692b3503aef02f1d865942, REOPEN/MOVE}] 2023-04-15 02:55:04,496 INFO [PEWorker-4] procedure.MasterProcedureScheduler(727): Took xlock for pid=40, ppid=39, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=104a1f3a98692b3503aef02f1d865942, REOPEN/MOVE 2023-04-15 02:55:04,496 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=40 updating hbase:meta row=104a1f3a98692b3503aef02f1d865942, regionState=CLOSING, regionLocation=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:04,496 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1681527301940.104a1f3a98692b3503aef02f1d865942.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1681527304496"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1681527304496"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1681527304496"}]},"ts":"1681527304496"} 2023-04-15 02:55:04,497 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=41, ppid=40, state=RUNNABLE; CloseRegionProcedure 104a1f3a98692b3503aef02f1d865942, server=jenkins-hbase20.apache.org,34681,1681527209133}] 2023-04-15 02:55:04,649 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] handler.UnassignRegionHandler(109): Close 104a1f3a98692b3503aef02f1d865942 2023-04-15 02:55:04,650 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1604): Closing 104a1f3a98692b3503aef02f1d865942, disabling compactions & flushes 2023-04-15 02:55:04,650 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1681527301940.104a1f3a98692b3503aef02f1d865942. 2023-04-15 02:55:04,650 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1681527301940.104a1f3a98692b3503aef02f1d865942. 2023-04-15 02:55:04,650 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1681527301940.104a1f3a98692b3503aef02f1d865942. after waiting 0 ms 2023-04-15 02:55:04,650 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1681527301940.104a1f3a98692b3503aef02f1d865942. 2023-04-15 02:55:04,655 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/104a1f3a98692b3503aef02f1d865942/recovered.edits/4.seqid, newMaxSeqId=4, maxSeqId=1 2023-04-15 02:55:04,656 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1681527301940.104a1f3a98692b3503aef02f1d865942. 2023-04-15 02:55:04,656 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1558): Region close journal for 104a1f3a98692b3503aef02f1d865942: 2023-04-15 02:55:04,656 WARN [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionServer(3509): Not adding moved region record: 104a1f3a98692b3503aef02f1d865942 to self. 2023-04-15 02:55:04,658 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] handler.UnassignRegionHandler(147): Closed 104a1f3a98692b3503aef02f1d865942 2023-04-15 02:55:04,658 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=40 updating hbase:meta row=104a1f3a98692b3503aef02f1d865942, regionState=CLOSED 2023-04-15 02:55:04,658 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1681527301940.104a1f3a98692b3503aef02f1d865942.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1681527304658"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1681527304658"}]},"ts":"1681527304658"} 2023-04-15 02:55:04,661 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=41, resume processing ppid=40 2023-04-15 02:55:04,661 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=41, ppid=40, state=SUCCESS; CloseRegionProcedure 104a1f3a98692b3503aef02f1d865942, server=jenkins-hbase20.apache.org,34681,1681527209133 in 162 msec 2023-04-15 02:55:04,662 INFO [PEWorker-1] assignment.TransitRegionStateProcedure(193): Starting pid=40, ppid=39, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=TestAcidGuarantees, region=104a1f3a98692b3503aef02f1d865942, REOPEN/MOVE; state=CLOSED, location=jenkins-hbase20.apache.org,34681,1681527209133; forceNewPlan=false, retain=true 2023-04-15 02:55:04,812 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=40 updating hbase:meta row=104a1f3a98692b3503aef02f1d865942, regionState=OPENING, regionLocation=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:04,812 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1681527301940.104a1f3a98692b3503aef02f1d865942.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1681527304812"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1681527304812"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1681527304812"}]},"ts":"1681527304812"} 2023-04-15 02:55:04,814 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=42, ppid=40, state=RUNNABLE; OpenRegionProcedure 104a1f3a98692b3503aef02f1d865942, server=jenkins-hbase20.apache.org,34681,1681527209133}] 2023-04-15 02:55:04,969 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] handler.AssignRegionHandler(128): Open TestAcidGuarantees,,1681527301940.104a1f3a98692b3503aef02f1d865942. 2023-04-15 02:55:04,969 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => 104a1f3a98692b3503aef02f1d865942, NAME => 'TestAcidGuarantees,,1681527301940.104a1f3a98692b3503aef02f1d865942.', STARTKEY => '', ENDKEY => ''} 2023-04-15 02:55:04,970 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table TestAcidGuarantees 104a1f3a98692b3503aef02f1d865942 2023-04-15 02:55:04,970 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1681527301940.104a1f3a98692b3503aef02f1d865942.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-15 02:55:04,970 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7894): checking encryption for 104a1f3a98692b3503aef02f1d865942 2023-04-15 02:55:04,970 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7897): checking classloading for 104a1f3a98692b3503aef02f1d865942 2023-04-15 02:55:04,971 INFO [StoreOpener-104a1f3a98692b3503aef02f1d865942-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family A of region 104a1f3a98692b3503aef02f1d865942 2023-04-15 02:55:04,972 DEBUG [StoreOpener-104a1f3a98692b3503aef02f1d865942-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/104a1f3a98692b3503aef02f1d865942/A 2023-04-15 02:55:04,972 DEBUG [StoreOpener-104a1f3a98692b3503aef02f1d865942-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/104a1f3a98692b3503aef02f1d865942/A 2023-04-15 02:55:04,973 INFO [StoreOpener-104a1f3a98692b3503aef02f1d865942-1] regionserver.CompactingMemStore(121): Store=A, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-15 02:55:04,973 INFO [StoreOpener-104a1f3a98692b3503aef02f1d865942-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 104a1f3a98692b3503aef02f1d865942 columnFamilyName A 2023-04-15 02:55:04,975 INFO [StoreOpener-104a1f3a98692b3503aef02f1d865942-1] regionserver.HStore(310): Store=104a1f3a98692b3503aef02f1d865942/A, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-15 02:55:04,975 INFO [StoreOpener-104a1f3a98692b3503aef02f1d865942-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family B of region 104a1f3a98692b3503aef02f1d865942 2023-04-15 02:55:04,976 DEBUG [StoreOpener-104a1f3a98692b3503aef02f1d865942-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/104a1f3a98692b3503aef02f1d865942/B 2023-04-15 02:55:04,976 DEBUG [StoreOpener-104a1f3a98692b3503aef02f1d865942-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/104a1f3a98692b3503aef02f1d865942/B 2023-04-15 02:55:04,977 INFO [StoreOpener-104a1f3a98692b3503aef02f1d865942-1] regionserver.CompactingMemStore(121): Store=B, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-15 02:55:04,977 INFO [StoreOpener-104a1f3a98692b3503aef02f1d865942-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 104a1f3a98692b3503aef02f1d865942 columnFamilyName B 2023-04-15 02:55:04,978 INFO [StoreOpener-104a1f3a98692b3503aef02f1d865942-1] regionserver.HStore(310): Store=104a1f3a98692b3503aef02f1d865942/B, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-15 02:55:04,978 INFO [StoreOpener-104a1f3a98692b3503aef02f1d865942-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family C of region 104a1f3a98692b3503aef02f1d865942 2023-04-15 02:55:04,979 DEBUG [StoreOpener-104a1f3a98692b3503aef02f1d865942-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/104a1f3a98692b3503aef02f1d865942/C 2023-04-15 02:55:04,979 DEBUG [StoreOpener-104a1f3a98692b3503aef02f1d865942-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/104a1f3a98692b3503aef02f1d865942/C 2023-04-15 02:55:04,980 INFO [StoreOpener-104a1f3a98692b3503aef02f1d865942-1] regionserver.CompactingMemStore(121): Store=C, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-15 02:55:04,980 INFO [StoreOpener-104a1f3a98692b3503aef02f1d865942-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 104a1f3a98692b3503aef02f1d865942 columnFamilyName C 2023-04-15 02:55:04,980 INFO [StoreOpener-104a1f3a98692b3503aef02f1d865942-1] regionserver.HStore(310): Store=104a1f3a98692b3503aef02f1d865942/C, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-15 02:55:04,981 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1146): Setting FlushNonSloppyStoresFirstPolicy for the region=TestAcidGuarantees,,1681527301940.104a1f3a98692b3503aef02f1d865942. 2023-04-15 02:55:04,981 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/104a1f3a98692b3503aef02f1d865942 2023-04-15 02:55:04,983 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/104a1f3a98692b3503aef02f1d865942 2023-04-15 02:55:04,985 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table TestAcidGuarantees descriptor;using region.getMemStoreFlushHeapSize/# of families (16.0 M)) instead. 2023-04-15 02:55:04,987 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1055): writing seq id for 104a1f3a98692b3503aef02f1d865942 2023-04-15 02:55:04,988 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1072): Opened 104a1f3a98692b3503aef02f1d865942; next sequenceid=5; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=73254533, jitterRate=0.09157760441303253}, FlushLargeStoresPolicy{flushSizeLowerBound=16777216} 2023-04-15 02:55:04,988 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(965): Region open journal for 104a1f3a98692b3503aef02f1d865942: 2023-04-15 02:55:04,989 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for TestAcidGuarantees,,1681527301940.104a1f3a98692b3503aef02f1d865942., pid=42, masterSystemTime=1681527304966 2023-04-15 02:55:04,991 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for TestAcidGuarantees,,1681527301940.104a1f3a98692b3503aef02f1d865942. 2023-04-15 02:55:04,991 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] handler.AssignRegionHandler(156): Opened TestAcidGuarantees,,1681527301940.104a1f3a98692b3503aef02f1d865942. 2023-04-15 02:55:04,991 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=40 updating hbase:meta row=104a1f3a98692b3503aef02f1d865942, regionState=OPEN, openSeqNum=5, regionLocation=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:04,992 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"TestAcidGuarantees,,1681527301940.104a1f3a98692b3503aef02f1d865942.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1681527304991"},{"qualifier":"server","vlen":32,"tag":[],"timestamp":"1681527304991"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1681527304991"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1681527304991"}]},"ts":"1681527304991"} 2023-04-15 02:55:04,994 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=42, resume processing ppid=40 2023-04-15 02:55:04,995 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=42, ppid=40, state=SUCCESS; OpenRegionProcedure 104a1f3a98692b3503aef02f1d865942, server=jenkins-hbase20.apache.org,34681,1681527209133 in 179 msec 2023-04-15 02:55:04,996 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=40, resume processing ppid=39 2023-04-15 02:55:04,997 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=40, ppid=39, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=104a1f3a98692b3503aef02f1d865942, REOPEN/MOVE in 500 msec 2023-04-15 02:55:04,999 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=39, resume processing ppid=38 2023-04-15 02:55:05,000 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=39, ppid=38, state=SUCCESS; ReopenTableRegionsProcedure table=TestAcidGuarantees in 506 msec 2023-04-15 02:55:05,002 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=38, state=SUCCESS; ModifyTableProcedure table=TestAcidGuarantees in 934 msec 2023-04-15 02:55:05,003 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(1227): Checking to see if procedure is done pid=38 2023-04-15 02:55:05,005 DEBUG [Listener at localhost.localdomain/33693] zookeeper.ReadOnlyZKClient(139): Connect 0x6fa3f5c5 to 127.0.0.1:51762 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-15 02:55:05,010 DEBUG [Listener at localhost.localdomain/33693] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@7cc9ff5, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-15 02:55:05,011 DEBUG [Listener at localhost.localdomain/33693] zookeeper.ReadOnlyZKClient(139): Connect 0x6ba43635 to 127.0.0.1:51762 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-15 02:55:05,024 DEBUG [Listener at localhost.localdomain/33693] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@1bd9eb59, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-15 02:55:05,024 DEBUG [Listener at localhost.localdomain/33693] zookeeper.ReadOnlyZKClient(139): Connect 0x377ba302 to 127.0.0.1:51762 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-15 02:55:05,032 DEBUG [Listener at localhost.localdomain/33693] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@7d243e46, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-15 02:55:05,033 DEBUG [Listener at localhost.localdomain/33693] zookeeper.ReadOnlyZKClient(139): Connect 0x1eaa86ba to 127.0.0.1:51762 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-15 02:55:05,039 DEBUG [Listener at localhost.localdomain/33693] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@74a355de, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-15 02:55:05,039 DEBUG [Listener at localhost.localdomain/33693] zookeeper.ReadOnlyZKClient(139): Connect 0x1d63a8ce to 127.0.0.1:51762 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-15 02:55:05,049 DEBUG [Listener at localhost.localdomain/33693] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@57b6628e, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-15 02:55:05,050 DEBUG [Listener at localhost.localdomain/33693] zookeeper.ReadOnlyZKClient(139): Connect 0x5da5030e to 127.0.0.1:51762 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-15 02:55:05,057 DEBUG [Listener at localhost.localdomain/33693] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@39e3a826, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-15 02:55:05,057 DEBUG [Listener at localhost.localdomain/33693] zookeeper.ReadOnlyZKClient(139): Connect 0x74d6bf18 to 127.0.0.1:51762 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-15 02:55:05,064 DEBUG [Listener at localhost.localdomain/33693] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@19e7fff, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-15 02:55:05,064 DEBUG [Listener at localhost.localdomain/33693] zookeeper.ReadOnlyZKClient(139): Connect 0x060c5cba to 127.0.0.1:51762 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-15 02:55:05,074 DEBUG [Listener at localhost.localdomain/33693] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@47e30d1c, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-15 02:55:05,074 DEBUG [Listener at localhost.localdomain/33693] zookeeper.ReadOnlyZKClient(139): Connect 0x52249391 to 127.0.0.1:51762 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-15 02:55:05,082 DEBUG [Listener at localhost.localdomain/33693] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@36a9ece3, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-15 02:55:05,082 DEBUG [Listener at localhost.localdomain/33693] zookeeper.ReadOnlyZKClient(139): Connect 0x5ba3ce53 to 127.0.0.1:51762 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-15 02:55:05,089 DEBUG [Listener at localhost.localdomain/33693] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@2ded66ad, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-15 02:55:05,099 DEBUG [hconnection-0x5095f84f-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-15 02:55:05,100 DEBUG [hconnection-0x70a08197-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-15 02:55:05,100 DEBUG [hconnection-0x16e93c6d-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-15 02:55:05,100 DEBUG [hconnection-0xb0c92b1-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-15 02:55:05,100 DEBUG [hconnection-0xe264b78-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-15 02:55:05,102 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-15 02:55:05,106 DEBUG [hconnection-0x1f88af86-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-15 02:55:05,107 DEBUG [hconnection-0x1c2c998b-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-15 02:55:05,107 DEBUG [hconnection-0x2d3a968a-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-15 02:55:05,107 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:41128, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-15 02:55:05,107 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:41120, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-15 02:55:05,107 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:41118, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-15 02:55:05,108 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:41140, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-15 02:55:05,108 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:41164, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-15 02:55:05,112 DEBUG [hconnection-0x2f5a6f82-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-15 02:55:05,112 DEBUG [hconnection-0x5ebd9e98-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-15 02:55:05,113 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:41166, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-15 02:55:05,115 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:41156, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-15 02:55:05,115 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:41180, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-15 02:55:05,121 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:41186, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-15 02:55:05,122 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-15 02:55:05,122 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-15 02:55:05,122 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:41192, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-15 02:55:05,122 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-15 02:55:05,122 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-15 02:55:05,122 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-15 02:55:05,122 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-15 02:55:05,129 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:05,129 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:55:05,130 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-15 02:55:05,130 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-15 02:55:05,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-15 02:55:05,130 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:05,133 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:05,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:55:05,133 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-15 02:55:05,133 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34681] regionserver.HRegion(9158): Flush requested on 104a1f3a98692b3503aef02f1d865942 2023-04-15 02:55:05,133 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 104a1f3a98692b3503aef02f1d865942 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-15 02:55:05,134 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 104a1f3a98692b3503aef02f1d865942, store=A 2023-04-15 02:55:05,134 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-15 02:55:05,134 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 104a1f3a98692b3503aef02f1d865942, store=B 2023-04-15 02:55:05,134 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-15 02:55:05,134 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 104a1f3a98692b3503aef02f1d865942, store=C 2023-04-15 02:55:05,134 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-15 02:55:05,138 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:05,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-15 02:55:05,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:55:05,138 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-15 02:55:05,139 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-15 02:55:05,139 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-15 02:55:05,139 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-15 02:55:05,139 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-15 02:55:05,139 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-15 02:55:05,140 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-15 02:55:05,140 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-15 02:55:05,140 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681527301940.104a1f3a98692b3503aef02f1d865942. 2023-04-15 02:55:05,140 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681527301940.104a1f3a98692b3503aef02f1d865942. started... 2023-04-15 02:55:05,140 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681527301940.104a1f3a98692b3503aef02f1d865942. as already flushing 2023-04-15 02:55:05,140 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681527301940.104a1f3a98692b3503aef02f1d865942. 2023-04-15 02:55:05,140 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-15 02:55:05,140 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-15 02:55:05,140 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:05,142 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-15 02:55:05,142 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34681,1681527209133' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-15 02:55:05,143 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:05,144 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:05,144 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-15 02:55:05,144 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:05,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:05,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:55:05,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:55:05,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:55:05,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:55:05,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:55:05,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:05,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:55:05,146 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34681,1681527209133' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-15 02:55:05,146 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-15 02:55:05,146 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4645bd67[Count = 0] remaining members to acquire global barrier 2023-04-15 02:55:05,146 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:05,146 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:05,146 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:05,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:05,147 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-15 02:55:05,147 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-15 02:55:05,147 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34681,1681527209133' in zk 2023-04-15 02:55:05,147 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:05,147 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-15 02:55:05,148 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:05,148 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-15 02:55:05,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:05,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:55:05,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:55:05,148 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-15 02:55:05,148 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-15 02:55:05,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:55:05,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:55:05,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:55:05,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:05,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:55:05,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:55:05,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:05,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34681,1681527209133': 2023-04-15 02:55:05,152 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34681,1681527209133' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-15 02:55:05,152 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-15 02:55:05,152 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-15 02:55:05,152 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-15 02:55:05,152 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-15 02:55:05,152 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-15 02:55:05,155 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:05,155 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:05,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:05,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:55:05,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:55:05,155 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-15 02:55:05,155 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:05,155 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:05,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:05,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:55:05,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:55:05,156 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-15 02:55:05,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-15 02:55:05,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:55:05,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:05,156 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:05,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:55:05,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:05,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:55:05,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:55:05,159 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:05,159 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-15 02:55:05,159 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-15 02:55:05,159 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:55:05,159 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-15 02:55:05,160 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-15 02:55:05,160 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-15 02:55:05,160 DEBUG [Thread-2181] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-15 02:55:05,159 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:55:05,159 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-15 02:55:05,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-15 02:55:05,159 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-15 02:55:05,160 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:05,160 DEBUG [Thread-2181] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-15 02:55:05,160 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:05,160 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:05,160 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-15 02:55:05,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-15 02:55:05,160 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:05,167 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=104a1f3a98692b3503aef02f1d865942, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:55:05,167 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34681] ipc.CallRunner(144): callId: 6 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41118 deadline: 1681527365159, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=104a1f3a98692b3503aef02f1d865942, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:05,167 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=104a1f3a98692b3503aef02f1d865942, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:55:05,167 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34681] ipc.CallRunner(144): callId: 8 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41164 deadline: 1681527365160, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=104a1f3a98692b3503aef02f1d865942, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:05,168 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1681527305143, max timestamp=1681527305166Num uniques 859; 2023-04-15 02:55:05,168 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-3] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-04-15 02:55:05,169 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1681527305143, max timestamp=1681527305166Num uniques 859; 2023-04-15 02:55:05,170 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-3] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-04-15 02:55:05,171 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1681527305143, max timestamp=1681527305166Num uniques 859; 2023-04-15 02:55:05,171 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-3] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-04-15 02:55:05,182 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=104a1f3a98692b3503aef02f1d865942, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:55:05,182 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34681] ipc.CallRunner(144): callId: 13 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41156 deadline: 1681527365180, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=104a1f3a98692b3503aef02f1d865942, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:05,182 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=104a1f3a98692b3503aef02f1d865942, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:55:05,182 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34681] ipc.CallRunner(144): callId: 13 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41180 deadline: 1681527365182, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=104a1f3a98692b3503aef02f1d865942, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:05,182 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=104a1f3a98692b3503aef02f1d865942, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:55:05,182 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34681] ipc.CallRunner(144): callId: 10 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41166 deadline: 1681527365182, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=104a1f3a98692b3503aef02f1d865942, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:05,260 DEBUG [Thread-2181] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-15 02:55:05,261 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-15 02:55:05,263 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-15 02:55:05,270 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=104a1f3a98692b3503aef02f1d865942, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:55:05,270 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34681] ipc.CallRunner(144): callId: 8 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41118 deadline: 1681527365270, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=104a1f3a98692b3503aef02f1d865942, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:05,273 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=104a1f3a98692b3503aef02f1d865942, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:55:05,273 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34681] ipc.CallRunner(144): callId: 10 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41164 deadline: 1681527365273, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=104a1f3a98692b3503aef02f1d865942, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:05,281 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-15 02:55:05,281 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-15 02:55:05,281 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-15 02:55:05,281 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-15 02:55:05,281 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-15 02:55:05,281 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-15 02:55:05,282 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:05,282 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:55:05,284 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:05,284 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-15 02:55:05,284 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-15 02:55:05,284 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:05,284 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-15 02:55:05,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-15 02:55:05,287 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=104a1f3a98692b3503aef02f1d865942, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:55:05,288 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34681] ipc.CallRunner(144): callId: 15 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41180 deadline: 1681527365283, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=104a1f3a98692b3503aef02f1d865942, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:05,288 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=104a1f3a98692b3503aef02f1d865942, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:55:05,288 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34681] ipc.CallRunner(144): callId: 12 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41166 deadline: 1681527365284, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=104a1f3a98692b3503aef02f1d865942, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:05,288 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=104a1f3a98692b3503aef02f1d865942, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:55:05,288 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34681] ipc.CallRunner(144): callId: 15 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41156 deadline: 1681527365285, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=104a1f3a98692b3503aef02f1d865942, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:05,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:55:05,295 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:05,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-15 02:55:05,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:55:05,303 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-15 02:55:05,303 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-15 02:55:05,303 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-15 02:55:05,303 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-15 02:55:05,303 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-15 02:55:05,303 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-15 02:55:05,304 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-15 02:55:05,304 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-15 02:55:05,304 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681527301940.104a1f3a98692b3503aef02f1d865942. 2023-04-15 02:55:05,304 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681527301940.104a1f3a98692b3503aef02f1d865942. started... 2023-04-15 02:55:05,304 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681527301940.104a1f3a98692b3503aef02f1d865942. as already flushing 2023-04-15 02:55:05,304 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681527301940.104a1f3a98692b3503aef02f1d865942. 2023-04-15 02:55:05,304 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-15 02:55:05,304 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-15 02:55:05,304 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:05,304 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-15 02:55:05,304 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34681,1681527209133' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-15 02:55:05,308 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:05,308 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:05,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:05,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:55:05,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:55:05,309 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:05,309 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-15 02:55:05,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:55:05,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:55:05,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:55:05,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:05,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:55:05,310 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34681,1681527209133' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-15 02:55:05,310 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-15 02:55:05,310 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:05,310 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@875ee4b[Count = 0] remaining members to acquire global barrier 2023-04-15 02:55:05,312 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:05,312 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:05,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:05,312 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-15 02:55:05,312 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-15 02:55:05,312 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34681,1681527209133' in zk 2023-04-15 02:55:05,313 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:05,313 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-15 02:55:05,313 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-15 02:55:05,313 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-15 02:55:05,313 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-15 02:55:05,314 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:05,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:05,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:55:05,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:55:05,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:55:05,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:55:05,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:55:05,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:05,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:55:05,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:55:05,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:05,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34681,1681527209133': 2023-04-15 02:55:05,317 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34681,1681527209133' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-15 02:55:05,318 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-15 02:55:05,318 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-15 02:55:05,318 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-15 02:55:05,318 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-15 02:55:05,318 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-15 02:55:05,319 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:05,319 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-15 02:55:05,319 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:05,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:05,319 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:05,319 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-15 02:55:05,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-15 02:55:05,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:05,321 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:05,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:05,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:55:05,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:55:05,322 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:05,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:55:05,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:55:05,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:55:05,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:55:05,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:05,331 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-15 02:55:05,331 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-15 02:55:05,331 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-15 02:55:05,332 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:05,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-15 02:55:05,332 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-15 02:55:05,332 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-15 02:55:05,332 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-15 02:55:05,332 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:55:05,332 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:55:05,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:55:05,332 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-15 02:55:05,332 DEBUG [Thread-2181] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-15 02:55:05,332 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:05,332 DEBUG [Thread-2181] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-15 02:55:05,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-15 02:55:05,332 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:05,332 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:05,332 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:05,333 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-15 02:55:05,432 DEBUG [Thread-2181] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-15 02:55:05,433 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-15 02:55:05,434 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-15 02:55:05,442 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-15 02:55:05,442 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-15 02:55:05,442 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-15 02:55:05,442 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-15 02:55:05,442 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-15 02:55:05,442 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-15 02:55:05,443 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:05,443 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:55:05,443 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:05,443 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-15 02:55:05,444 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-15 02:55:05,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-15 02:55:05,444 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:05,444 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-15 02:55:05,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:55:05,444 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:05,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-15 02:55:05,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:55:05,445 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-15 02:55:05,462 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-15 02:55:05,462 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-15 02:55:05,462 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-15 02:55:05,462 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-15 02:55:05,462 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-15 02:55:05,466 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-15 02:55:05,466 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-15 02:55:05,467 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681527301940.104a1f3a98692b3503aef02f1d865942. 2023-04-15 02:55:05,467 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681527301940.104a1f3a98692b3503aef02f1d865942. started... 2023-04-15 02:55:05,467 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681527301940.104a1f3a98692b3503aef02f1d865942. as already flushing 2023-04-15 02:55:05,467 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681527301940.104a1f3a98692b3503aef02f1d865942. 2023-04-15 02:55:05,467 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-15 02:55:05,467 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-15 02:55:05,467 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:05,467 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-15 02:55:05,467 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34681,1681527209133' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-15 02:55:05,469 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:05,469 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:05,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:05,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:55:05,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:55:05,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:55:05,470 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:05,470 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-15 02:55:05,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:55:05,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:55:05,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:05,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:55:05,471 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34681,1681527209133' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-15 02:55:05,471 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-15 02:55:05,471 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@324b6c77[Count = 0] remaining members to acquire global barrier 2023-04-15 02:55:05,471 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:05,472 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:05,472 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:05,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:05,472 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-15 02:55:05,472 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:05,472 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-15 02:55:05,472 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-15 02:55:05,473 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34681,1681527209133' in zk 2023-04-15 02:55:05,478 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-15 02:55:05,478 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-15 02:55:05,478 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:05,478 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-15 02:55:05,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:05,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:55:05,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:55:05,479 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=104a1f3a98692b3503aef02f1d865942, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:55:05,479 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34681] ipc.CallRunner(144): callId: 10 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41118 deadline: 1681527365473, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=104a1f3a98692b3503aef02f1d865942, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:05,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:55:05,480 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=104a1f3a98692b3503aef02f1d865942, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:55:05,480 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34681] ipc.CallRunner(144): callId: 12 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41164 deadline: 1681527365475, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=104a1f3a98692b3503aef02f1d865942, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:05,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:55:05,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:55:05,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:05,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:55:05,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:55:05,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:05,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34681,1681527209133': 2023-04-15 02:55:05,482 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34681,1681527209133' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-15 02:55:05,482 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-15 02:55:05,482 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-15 02:55:05,482 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-15 02:55:05,482 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-15 02:55:05,482 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-15 02:55:05,489 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=104a1f3a98692b3503aef02f1d865942, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:55:05,489 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34681] ipc.CallRunner(144): callId: 17 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41180 deadline: 1681527365489, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=104a1f3a98692b3503aef02f1d865942, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:05,490 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:05,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:05,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:55:05,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:55:05,491 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:05,491 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-15 02:55:05,491 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:05,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:05,492 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-15 02:55:05,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-15 02:55:05,492 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=104a1f3a98692b3503aef02f1d865942, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:55:05,492 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=104a1f3a98692b3503aef02f1d865942, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:55:05,492 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34681] ipc.CallRunner(144): callId: 14 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41166 deadline: 1681527365491, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=104a1f3a98692b3503aef02f1d865942, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:05,492 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34681] ipc.CallRunner(144): callId: 17 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41156 deadline: 1681527365492, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=104a1f3a98692b3503aef02f1d865942, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:05,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:05,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:55:05,492 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:05,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:55:05,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:55:05,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:55:05,494 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:05,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:05,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:55:05,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:55:05,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:05,498 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-15 02:55:05,498 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-15 02:55:05,498 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-15 02:55:05,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-15 02:55:05,498 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:05,498 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:55:05,498 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:55:05,498 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:05,498 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:05,498 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:05,498 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:05,499 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-15 02:55:05,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-15 02:55:05,499 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-15 02:55:05,499 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-15 02:55:05,499 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-15 02:55:05,499 DEBUG [Thread-2181] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-15 02:55:05,499 DEBUG [Thread-2181] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-15 02:55:05,560 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230415f0e979f2cca84ff9a35610e1f768de16 to hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230415f0e979f2cca84ff9a35610e1f768de16 2023-04-15 02:55:05,572 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=16, memsize=20.1 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/104a1f3a98692b3503aef02f1d865942/.tmp/A/032ee30bdfac405b91c9b37d38512266 2023-04-15 02:55:05,599 DEBUG [Thread-2181] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-15 02:55:05,600 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-15 02:55:05,601 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-15 02:55:05,610 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-15 02:55:05,610 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-15 02:55:05,610 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-15 02:55:05,610 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-15 02:55:05,610 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-15 02:55:05,610 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-15 02:55:05,610 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:05,611 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:55:05,611 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-15 02:55:05,611 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-15 02:55:05,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-15 02:55:05,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:55:05,612 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:05,612 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:05,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-15 02:55:05,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:55:05,612 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:05,612 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-15 02:55:05,612 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-15 02:55:05,633 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-15 02:55:05,633 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-15 02:55:05,633 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-15 02:55:05,633 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-15 02:55:05,633 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-15 02:55:05,650 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-15 02:55:05,650 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-15 02:55:05,650 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681527301940.104a1f3a98692b3503aef02f1d865942. 2023-04-15 02:55:05,650 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681527301940.104a1f3a98692b3503aef02f1d865942. started... 2023-04-15 02:55:05,650 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681527301940.104a1f3a98692b3503aef02f1d865942. as already flushing 2023-04-15 02:55:05,650 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681527301940.104a1f3a98692b3503aef02f1d865942. 2023-04-15 02:55:05,650 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-15 02:55:05,650 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-15 02:55:05,650 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:05,651 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-15 02:55:05,651 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34681,1681527209133' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-15 02:55:05,652 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:05,652 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:05,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:05,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:55:05,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:55:05,652 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:05,652 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-15 02:55:05,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:55:05,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:55:05,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:55:05,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:05,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:55:05,654 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34681,1681527209133' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-15 02:55:05,654 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1e417c9a[Count = 0] remaining members to acquire global barrier 2023-04-15 02:55:05,654 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-15 02:55:05,654 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:05,655 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:05,655 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-15 02:55:05,655 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:05,655 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:05,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:05,656 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-15 02:55:05,656 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-15 02:55:05,656 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34681,1681527209133' in zk 2023-04-15 02:55:05,658 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-15 02:55:05,658 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:05,658 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-15 02:55:05,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:05,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:55:05,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:55:05,658 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-15 02:55:05,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:55:05,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:55:05,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:55:05,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:05,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:55:05,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:55:05,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:05,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34681,1681527209133': 2023-04-15 02:55:05,661 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34681,1681527209133' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-15 02:55:05,661 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-15 02:55:05,661 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-15 02:55:05,661 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-15 02:55:05,661 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-15 02:55:05,661 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-15 02:55:05,662 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:05,662 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:05,663 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-15 02:55:05,663 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:05,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:05,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:05,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:55:05,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:55:05,663 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:05,663 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-15 02:55:05,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-15 02:55:05,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:05,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:55:05,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:55:05,664 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:05,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:55:05,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:55:05,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:05,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:55:05,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:55:05,667 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:05,667 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:55:05,667 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:55:05,667 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-15 02:55:05,667 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-15 02:55:05,667 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-15 02:55:05,667 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-15 02:55:05,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-15 02:55:05,667 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-15 02:55:05,667 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-15 02:55:05,667 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-15 02:55:05,668 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:05,668 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:05,668 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:05,668 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:05,668 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-15 02:55:05,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-15 02:55:05,671 DEBUG [Thread-2181] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-15 02:55:05,671 DEBUG [Thread-2181] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-15 02:55:05,772 DEBUG [Thread-2181] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-15 02:55:05,772 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-15 02:55:05,776 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-15 02:55:05,784 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=104a1f3a98692b3503aef02f1d865942, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:55:05,784 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34681] ipc.CallRunner(144): callId: 12 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41118 deadline: 1681527365781, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=104a1f3a98692b3503aef02f1d865942, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:05,784 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=104a1f3a98692b3503aef02f1d865942, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:55:05,784 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34681] ipc.CallRunner(144): callId: 14 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41164 deadline: 1681527365782, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=104a1f3a98692b3503aef02f1d865942, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:05,794 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-15 02:55:05,794 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-15 02:55:05,794 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-15 02:55:05,794 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-15 02:55:05,794 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-15 02:55:05,794 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-15 02:55:05,794 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=104a1f3a98692b3503aef02f1d865942, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:55:05,794 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34681] ipc.CallRunner(144): callId: 19 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41180 deadline: 1681527365793, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=104a1f3a98692b3503aef02f1d865942, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:05,794 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:05,795 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=104a1f3a98692b3503aef02f1d865942, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:55:05,795 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:55:05,795 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34681] ipc.CallRunner(144): callId: 19 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41156 deadline: 1681527365793, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=104a1f3a98692b3503aef02f1d865942, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:05,795 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:05,795 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-15 02:55:05,796 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-15 02:55:05,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-15 02:55:05,796 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:05,796 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-15 02:55:05,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:55:05,796 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:05,797 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=104a1f3a98692b3503aef02f1d865942, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:55:05,797 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34681] ipc.CallRunner(144): callId: 16 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41166 deadline: 1681527365794, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=104a1f3a98692b3503aef02f1d865942, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:05,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-15 02:55:05,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:55:05,798 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-15 02:55:05,804 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-15 02:55:05,804 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-15 02:55:05,804 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-15 02:55:05,804 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-15 02:55:05,804 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-15 02:55:05,806 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-15 02:55:05,806 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-15 02:55:05,806 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681527301940.104a1f3a98692b3503aef02f1d865942. 2023-04-15 02:55:05,806 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681527301940.104a1f3a98692b3503aef02f1d865942. started... 2023-04-15 02:55:05,806 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681527301940.104a1f3a98692b3503aef02f1d865942. as already flushing 2023-04-15 02:55:05,806 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681527301940.104a1f3a98692b3503aef02f1d865942. 2023-04-15 02:55:05,806 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-15 02:55:05,806 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-15 02:55:05,806 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:05,806 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-15 02:55:05,806 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34681,1681527209133' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-15 02:55:05,816 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:05,816 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:05,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:05,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:55:05,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:55:05,816 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:05,816 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-15 02:55:05,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:55:05,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:55:05,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:55:05,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:05,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:55:05,818 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34681,1681527209133' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-15 02:55:05,818 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@758454b0[Count = 0] remaining members to acquire global barrier 2023-04-15 02:55:05,818 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-15 02:55:05,818 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:05,822 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:05,822 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:05,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:05,822 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-15 02:55:05,822 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-15 02:55:05,822 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34681,1681527209133' in zk 2023-04-15 02:55:05,822 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:05,822 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-15 02:55:05,823 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:05,823 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-15 02:55:05,823 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-15 02:55:05,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:05,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:55:05,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:55:05,823 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-15 02:55:05,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:55:05,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:55:05,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:55:05,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:05,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:55:05,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:55:05,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:05,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34681,1681527209133': 2023-04-15 02:55:05,827 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34681,1681527209133' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-15 02:55:05,827 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-15 02:55:05,827 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-15 02:55:05,827 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-15 02:55:05,827 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-15 02:55:05,827 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-15 02:55:05,831 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:05,831 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-15 02:55:05,831 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:05,831 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:05,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:05,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:05,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:55:05,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:55:05,832 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-15 02:55:05,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-15 02:55:05,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:05,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:55:05,832 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:05,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:55:05,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:55:05,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:55:05,835 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:05,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:05,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:55:05,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:55:05,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:05,840 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-15 02:55:05,840 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-15 02:55:05,840 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-15 02:55:05,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-15 02:55:05,840 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:05,840 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:55:05,840 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:55:05,840 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:05,840 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:05,840 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-15 02:55:05,840 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-15 02:55:05,840 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-15 02:55:05,840 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:05,840 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:05,840 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-15 02:55:05,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-15 02:55:05,840 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-15 02:55:05,841 DEBUG [Thread-2181] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-15 02:55:05,841 DEBUG [Thread-2181] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-15 02:55:05,941 DEBUG [Thread-2181] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-15 02:55:05,942 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-15 02:55:05,944 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-15 02:55:05,946 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-15 02:55:05,946 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-15 02:55:05,946 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-15 02:55:05,946 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-15 02:55:05,946 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-15 02:55:05,946 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-15 02:55:05,946 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:05,947 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:55:05,947 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:05,947 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-15 02:55:05,947 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:05,947 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-15 02:55:05,948 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-15 02:55:05,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-15 02:55:05,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:55:05,948 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:05,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-15 02:55:05,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:55:05,949 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-15 02:55:05,958 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-15 02:55:05,958 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-15 02:55:05,958 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-15 02:55:05,958 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-15 02:55:05,958 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-15 02:55:05,959 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-15 02:55:05,959 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-15 02:55:05,959 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681527301940.104a1f3a98692b3503aef02f1d865942. 2023-04-15 02:55:05,959 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681527301940.104a1f3a98692b3503aef02f1d865942. started... 2023-04-15 02:55:05,959 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681527301940.104a1f3a98692b3503aef02f1d865942. as already flushing 2023-04-15 02:55:05,959 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681527301940.104a1f3a98692b3503aef02f1d865942. 2023-04-15 02:55:05,959 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-15 02:55:05,959 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-15 02:55:05,959 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:05,959 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-15 02:55:05,959 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34681,1681527209133' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-15 02:55:05,963 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:05,963 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:05,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:05,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:55:05,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:55:05,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:55:05,964 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:05,964 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-15 02:55:05,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:55:05,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:55:05,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:05,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:55:05,965 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34681,1681527209133' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-15 02:55:05,965 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@52a4ebfd[Count = 0] remaining members to acquire global barrier 2023-04-15 02:55:05,965 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-15 02:55:05,965 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:05,966 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:05,966 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:05,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:05,966 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-15 02:55:05,966 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-15 02:55:05,966 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34681,1681527209133' in zk 2023-04-15 02:55:05,966 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:05,966 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-15 02:55:05,967 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-15 02:55:05,967 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:05,967 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-15 02:55:05,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:05,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:55:05,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:55:05,967 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-15 02:55:05,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:55:05,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:55:05,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:55:05,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:05,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:55:05,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:55:05,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:05,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34681,1681527209133': 2023-04-15 02:55:05,970 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34681,1681527209133' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-15 02:55:05,970 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-15 02:55:05,970 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-15 02:55:05,970 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-15 02:55:05,970 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-15 02:55:05,970 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-15 02:55:05,971 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:05,971 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:05,971 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:05,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:05,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:05,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:55:05,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:55:05,971 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-15 02:55:05,971 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-15 02:55:05,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-15 02:55:05,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:05,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:55:05,971 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:05,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:55:05,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:55:05,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:55:05,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:05,973 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:05,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:55:05,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:55:05,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:05,974 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:05,974 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-15 02:55:05,974 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-15 02:55:05,974 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:55:05,974 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-15 02:55:05,974 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-15 02:55:05,974 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-15 02:55:05,974 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-15 02:55:05,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-15 02:55:05,974 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:55:05,975 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:05,975 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:05,975 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:05,975 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:05,975 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-15 02:55:05,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-15 02:55:05,975 DEBUG [Thread-2181] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-15 02:55:05,975 DEBUG [Thread-2181] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-15 02:55:06,075 DEBUG [Thread-2181] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-15 02:55:06,075 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-15 02:55:06,079 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-15 02:55:06,081 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=16 (bloomFilter=true), to=hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/104a1f3a98692b3503aef02f1d865942/.tmp/B/697475e420914a2d85b9414f5bdfcb6e 2023-04-15 02:55:06,090 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-15 02:55:06,090 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-15 02:55:06,090 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-15 02:55:06,090 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-15 02:55:06,090 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-15 02:55:06,090 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-15 02:55:06,091 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:06,091 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:55:06,092 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-15 02:55:06,092 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-15 02:55:06,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-15 02:55:06,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:55:06,096 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:06,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-15 02:55:06,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:55:06,096 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-15 02:55:06,099 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:06,101 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:06,101 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-15 02:55:06,101 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-15 02:55:06,101 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-15 02:55:06,101 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-15 02:55:06,101 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-15 02:55:06,102 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-15 02:55:06,102 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-15 02:55:06,102 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-15 02:55:06,102 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681527301940.104a1f3a98692b3503aef02f1d865942. 2023-04-15 02:55:06,102 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681527301940.104a1f3a98692b3503aef02f1d865942. started... 2023-04-15 02:55:06,102 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681527301940.104a1f3a98692b3503aef02f1d865942. as already flushing 2023-04-15 02:55:06,102 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681527301940.104a1f3a98692b3503aef02f1d865942. 2023-04-15 02:55:06,102 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-15 02:55:06,102 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-15 02:55:06,102 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:06,102 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-15 02:55:06,102 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34681,1681527209133' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-15 02:55:06,105 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:06,105 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:06,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:06,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:55:06,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:55:06,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:55:06,106 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:06,106 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-15 02:55:06,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:55:06,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:55:06,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:06,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:55:06,110 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34681,1681527209133' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-15 02:55:06,110 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3b651d0a[Count = 0] remaining members to acquire global barrier 2023-04-15 02:55:06,110 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-15 02:55:06,110 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:06,113 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:06,113 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:06,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:06,113 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-15 02:55:06,113 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-15 02:55:06,113 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34681,1681527209133' in zk 2023-04-15 02:55:06,114 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:06,114 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-15 02:55:06,115 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-15 02:55:06,115 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-15 02:55:06,115 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-15 02:55:06,115 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:06,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:06,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:55:06,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:55:06,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:55:06,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:55:06,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:55:06,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:06,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:55:06,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:55:06,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:06,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34681,1681527209133': 2023-04-15 02:55:06,118 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34681,1681527209133' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-15 02:55:06,118 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-15 02:55:06,118 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-15 02:55:06,118 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-15 02:55:06,118 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-15 02:55:06,118 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-15 02:55:06,119 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:06,119 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-15 02:55:06,119 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:06,119 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:06,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:06,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:06,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:55:06,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:55:06,119 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-15 02:55:06,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-15 02:55:06,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:06,120 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:06,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:55:06,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:55:06,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:55:06,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:55:06,122 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:06,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:06,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:55:06,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:55:06,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:06,124 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-15 02:55:06,124 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-15 02:55:06,124 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:06,124 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:55:06,124 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:55:06,124 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:06,124 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:06,124 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-15 02:55:06,124 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-15 02:55:06,125 DEBUG [Thread-2181] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-15 02:55:06,125 DEBUG [Thread-2181] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-15 02:55:06,124 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-15 02:55:06,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-15 02:55:06,124 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-15 02:55:06,124 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:06,125 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:06,125 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-15 02:55:06,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-15 02:55:06,124 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-15 02:55:06,128 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=16 (bloomFilter=true), to=hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/104a1f3a98692b3503aef02f1d865942/.tmp/C/c48c0a175cd04484a3922fce0950b506 2023-04-15 02:55:06,140 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/104a1f3a98692b3503aef02f1d865942/.tmp/A/032ee30bdfac405b91c9b37d38512266 as hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/104a1f3a98692b3503aef02f1d865942/A/032ee30bdfac405b91c9b37d38512266 2023-04-15 02:55:06,146 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/104a1f3a98692b3503aef02f1d865942/A/032ee30bdfac405b91c9b37d38512266, entries=100, sequenceid=16, filesize=18.4 K 2023-04-15 02:55:06,147 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/104a1f3a98692b3503aef02f1d865942/.tmp/B/697475e420914a2d85b9414f5bdfcb6e as hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/104a1f3a98692b3503aef02f1d865942/B/697475e420914a2d85b9414f5bdfcb6e 2023-04-15 02:55:06,153 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/104a1f3a98692b3503aef02f1d865942/B/697475e420914a2d85b9414f5bdfcb6e, entries=100, sequenceid=16, filesize=9.3 K 2023-04-15 02:55:06,154 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/104a1f3a98692b3503aef02f1d865942/.tmp/C/c48c0a175cd04484a3922fce0950b506 as hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/104a1f3a98692b3503aef02f1d865942/C/c48c0a175cd04484a3922fce0950b506 2023-04-15 02:55:06,160 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/104a1f3a98692b3503aef02f1d865942/C/c48c0a175cd04484a3922fce0950b506, entries=100, sequenceid=16, filesize=9.3 K 2023-04-15 02:55:06,161 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~60.38 KB/61830, heapSize ~180 KB/184320, currentSize=127.47 KB/130530 for 104a1f3a98692b3503aef02f1d865942 in 1028ms, sequenceid=16, compaction requested=false 2023-04-15 02:55:06,161 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 104a1f3a98692b3503aef02f1d865942: 2023-04-15 02:55:06,225 DEBUG [Thread-2181] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-15 02:55:06,226 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-15 02:55:06,228 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-15 02:55:06,238 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-15 02:55:06,238 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-15 02:55:06,238 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-15 02:55:06,238 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-15 02:55:06,238 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-15 02:55:06,238 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-15 02:55:06,239 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:06,239 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:55:06,239 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-15 02:55:06,239 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:06,239 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-15 02:55:06,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-15 02:55:06,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:55:06,240 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:06,240 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-15 02:55:06,240 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:06,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-15 02:55:06,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:55:06,241 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-15 02:55:06,245 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-15 02:55:06,245 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-15 02:55:06,245 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-15 02:55:06,245 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-15 02:55:06,245 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-15 02:55:06,253 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-15 02:55:06,281 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681527301940.104a1f3a98692b3503aef02f1d865942. 2023-04-15 02:55:06,281 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-15 02:55:06,281 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681527301940.104a1f3a98692b3503aef02f1d865942. started... 2023-04-15 02:55:06,282 INFO [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 104a1f3a98692b3503aef02f1d865942 3/3 column families, dataSize=127.47 KB heapSize=355.69 KB 2023-04-15 02:55:06,282 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 104a1f3a98692b3503aef02f1d865942, store=A 2023-04-15 02:55:06,282 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-15 02:55:06,282 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 104a1f3a98692b3503aef02f1d865942, store=B 2023-04-15 02:55:06,282 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-15 02:55:06,282 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 104a1f3a98692b3503aef02f1d865942, store=C 2023-04-15 02:55:06,282 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-15 02:55:06,291 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681527301940.104a1f3a98692b3503aef02f1d865942. as already flushing 2023-04-15 02:55:06,291 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34681] regionserver.HRegion(9158): Flush requested on 104a1f3a98692b3503aef02f1d865942 2023-04-15 02:55:06,314 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=104a1f3a98692b3503aef02f1d865942, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:55:06,314 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34681] ipc.CallRunner(144): callId: 18 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41166 deadline: 1681527366304, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=104a1f3a98692b3503aef02f1d865942, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:06,321 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=104a1f3a98692b3503aef02f1d865942, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:55:06,321 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34681] ipc.CallRunner(144): callId: 22 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41180 deadline: 1681527366311, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=104a1f3a98692b3503aef02f1d865942, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:06,322 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=104a1f3a98692b3503aef02f1d865942, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:55:06,322 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34681] ipc.CallRunner(144): callId: 22 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41156 deadline: 1681527366311, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=104a1f3a98692b3503aef02f1d865942, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:06,322 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=104a1f3a98692b3503aef02f1d865942, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:55:06,322 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34681] ipc.CallRunner(144): callId: 19 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41164 deadline: 1681527366312, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=104a1f3a98692b3503aef02f1d865942, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:06,328 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=104a1f3a98692b3503aef02f1d865942, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:55:06,328 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34681] ipc.CallRunner(144): callId: 17 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41118 deadline: 1681527366314, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=104a1f3a98692b3503aef02f1d865942, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:06,335 INFO [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230415ed72b9f131094c79bd7a1df22519caf6 to hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230415ed72b9f131094c79bd7a1df22519caf6 2023-04-15 02:55:06,373 INFO [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=50, memsize=42.5 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/104a1f3a98692b3503aef02f1d865942/.tmp/A/3cbb598a68f74c48bb9978ac9f332692 2023-04-15 02:55:06,417 INFO [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=50 (bloomFilter=true), to=hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/104a1f3a98692b3503aef02f1d865942/.tmp/B/ae3bbfafcdc747f996d9c62a7b80d13f 2023-04-15 02:55:06,425 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=104a1f3a98692b3503aef02f1d865942, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:55:06,425 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34681] ipc.CallRunner(144): callId: 24 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41180 deadline: 1681527366423, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=104a1f3a98692b3503aef02f1d865942, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:06,426 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=104a1f3a98692b3503aef02f1d865942, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:55:06,426 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34681] ipc.CallRunner(144): callId: 24 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41156 deadline: 1681527366423, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=104a1f3a98692b3503aef02f1d865942, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:06,429 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=104a1f3a98692b3503aef02f1d865942, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:55:06,429 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34681] ipc.CallRunner(144): callId: 21 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41164 deadline: 1681527366424, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=104a1f3a98692b3503aef02f1d865942, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:06,436 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=104a1f3a98692b3503aef02f1d865942, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:55:06,436 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34681] ipc.CallRunner(144): callId: 19 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41118 deadline: 1681527366429, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=104a1f3a98692b3503aef02f1d865942, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:06,446 WARN [HBase-Metrics2-1] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties 2023-04-15 02:55:06,501 INFO [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=50 (bloomFilter=true), to=hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/104a1f3a98692b3503aef02f1d865942/.tmp/C/93e4377e68b9479f839f5be916fb3d7f 2023-04-15 02:55:06,506 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/104a1f3a98692b3503aef02f1d865942/.tmp/A/3cbb598a68f74c48bb9978ac9f332692 as hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/104a1f3a98692b3503aef02f1d865942/A/3cbb598a68f74c48bb9978ac9f332692 2023-04-15 02:55:06,512 INFO [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/104a1f3a98692b3503aef02f1d865942/A/3cbb598a68f74c48bb9978ac9f332692, entries=150, sequenceid=50, filesize=25.1 K 2023-04-15 02:55:06,512 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/104a1f3a98692b3503aef02f1d865942/.tmp/B/ae3bbfafcdc747f996d9c62a7b80d13f as hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/104a1f3a98692b3503aef02f1d865942/B/ae3bbfafcdc747f996d9c62a7b80d13f 2023-04-15 02:55:06,527 INFO [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/104a1f3a98692b3503aef02f1d865942/B/ae3bbfafcdc747f996d9c62a7b80d13f, entries=150, sequenceid=50, filesize=11.6 K 2023-04-15 02:55:06,528 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/104a1f3a98692b3503aef02f1d865942/.tmp/C/93e4377e68b9479f839f5be916fb3d7f as hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/104a1f3a98692b3503aef02f1d865942/C/93e4377e68b9479f839f5be916fb3d7f 2023-04-15 02:55:06,534 INFO [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/104a1f3a98692b3503aef02f1d865942/C/93e4377e68b9479f839f5be916fb3d7f, entries=150, sequenceid=50, filesize=11.6 K 2023-04-15 02:55:06,535 INFO [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~127.47 KB/130530, heapSize ~355.64 KB/364176, currentSize=53.67 KB/54960 for 104a1f3a98692b3503aef02f1d865942 in 253ms, sequenceid=50, compaction requested=false 2023-04-15 02:55:06,535 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 104a1f3a98692b3503aef02f1d865942: 2023-04-15 02:55:06,535 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681527301940.104a1f3a98692b3503aef02f1d865942. 2023-04-15 02:55:06,535 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-15 02:55:06,535 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-15 02:55:06,535 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:06,535 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-15 02:55:06,535 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34681,1681527209133' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-15 02:55:06,538 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:06,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:06,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:55:06,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:55:06,538 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:06,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:55:06,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:55:06,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:55:06,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:06,541 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:06,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:55:06,541 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-15 02:55:06,542 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34681,1681527209133' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-15 02:55:06,542 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2ed50350[Count = 0] remaining members to acquire global barrier 2023-04-15 02:55:06,542 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-15 02:55:06,542 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:06,542 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:06,542 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:06,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:06,543 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-15 02:55:06,543 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-15 02:55:06,543 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34681,1681527209133' in zk 2023-04-15 02:55:06,543 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:06,543 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-15 02:55:06,544 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-15 02:55:06,544 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:06,544 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-15 02:55:06,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:06,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:55:06,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:55:06,544 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-15 02:55:06,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:55:06,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:55:06,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:55:06,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:06,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:55:06,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:55:06,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:06,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34681,1681527209133': 2023-04-15 02:55:06,547 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34681,1681527209133' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-15 02:55:06,547 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-15 02:55:06,547 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-15 02:55:06,547 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-15 02:55:06,547 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-15 02:55:06,547 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-15 02:55:06,548 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:06,548 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-15 02:55:06,548 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:06,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:06,549 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-15 02:55:06,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-15 02:55:06,549 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:06,549 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:06,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:06,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:55:06,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:06,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:55:06,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:55:06,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:55:06,551 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:06,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:55:06,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:55:06,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:06,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:55:06,554 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-15 02:55:06,554 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-15 02:55:06,554 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-15 02:55:06,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-15 02:55:06,554 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:06,555 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:55:06,555 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-15 02:55:06,555 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-15 02:55:06,555 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-15 02:55:06,555 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:55:06,555 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:06,555 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:06,555 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:06,555 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:06,555 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-15 02:55:06,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-15 02:55:06,555 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-15 02:55:06,555 DEBUG [Thread-2181] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-15 02:55:06,555 DEBUG [Thread-2181] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-15 02:55:06,630 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34681] regionserver.HRegion(9158): Flush requested on 104a1f3a98692b3503aef02f1d865942 2023-04-15 02:55:06,630 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 104a1f3a98692b3503aef02f1d865942 3/3 column families, dataSize=60.38 KB heapSize=180.05 KB 2023-04-15 02:55:06,630 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 104a1f3a98692b3503aef02f1d865942, store=A 2023-04-15 02:55:06,630 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-15 02:55:06,630 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 104a1f3a98692b3503aef02f1d865942, store=B 2023-04-15 02:55:06,630 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-15 02:55:06,630 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 104a1f3a98692b3503aef02f1d865942, store=C 2023-04-15 02:55:06,630 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-15 02:55:06,655 DEBUG [Thread-2181] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-15 02:55:06,656 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-15 02:55:06,657 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-15 02:55:06,658 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-15 02:55:06,658 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-15 02:55:06,658 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-15 02:55:06,658 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-15 02:55:06,658 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-15 02:55:06,658 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-15 02:55:06,659 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:06,659 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:55:06,659 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:06,659 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-15 02:55:06,659 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-15 02:55:06,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-15 02:55:06,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:55:06,660 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:06,660 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-15 02:55:06,660 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:06,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-15 02:55:06,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:55:06,661 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-15 02:55:06,662 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-15 02:55:06,662 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-15 02:55:06,662 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230415410f6eeb16544616a4b518ca971f0b1d to hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230415410f6eeb16544616a4b518ca971f0b1d 2023-04-15 02:55:06,666 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-15 02:55:06,666 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-15 02:55:06,666 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-15 02:55:06,673 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-15 02:55:06,673 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-15 02:55:06,673 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681527301940.104a1f3a98692b3503aef02f1d865942. 2023-04-15 02:55:06,674 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681527301940.104a1f3a98692b3503aef02f1d865942. started... 2023-04-15 02:55:06,674 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681527301940.104a1f3a98692b3503aef02f1d865942. as already flushing 2023-04-15 02:55:06,674 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681527301940.104a1f3a98692b3503aef02f1d865942. 2023-04-15 02:55:06,674 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-15 02:55:06,674 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-15 02:55:06,674 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:06,674 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-15 02:55:06,674 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34681,1681527209133' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-15 02:55:06,675 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:06,675 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:06,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:06,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:55:06,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:55:06,675 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:06,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:55:06,675 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-15 02:55:06,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:55:06,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:55:06,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:06,676 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=62, memsize=20.1 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/104a1f3a98692b3503aef02f1d865942/.tmp/A/4872b863c3ac49819160a8ded98bed42 2023-04-15 02:55:06,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:55:06,677 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34681,1681527209133' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-15 02:55:06,677 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-15 02:55:06,677 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:06,677 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@79a3e3ee[Count = 0] remaining members to acquire global barrier 2023-04-15 02:55:06,677 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:06,677 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:06,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:06,677 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-15 02:55:06,677 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-15 02:55:06,678 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34681,1681527209133' in zk 2023-04-15 02:55:06,678 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:06,678 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-15 02:55:06,679 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:06,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:06,679 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-15 02:55:06,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:55:06,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:55:06,679 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-15 02:55:06,679 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-15 02:55:06,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:55:06,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:55:06,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:55:06,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:06,680 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=104a1f3a98692b3503aef02f1d865942, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:55:06,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:55:06,681 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34681] ipc.CallRunner(144): callId: 24 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41118 deadline: 1681527366670, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=104a1f3a98692b3503aef02f1d865942, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:06,681 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=104a1f3a98692b3503aef02f1d865942, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:55:06,681 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34681] ipc.CallRunner(144): callId: 27 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41164 deadline: 1681527366672, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=104a1f3a98692b3503aef02f1d865942, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:06,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:55:06,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:06,683 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=104a1f3a98692b3503aef02f1d865942, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:55:06,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34681,1681527209133': 2023-04-15 02:55:06,683 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34681] ipc.CallRunner(144): callId: 32 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41180 deadline: 1681527366674, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=104a1f3a98692b3503aef02f1d865942, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:06,683 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-15 02:55:06,683 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-15 02:55:06,683 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-15 02:55:06,683 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34681,1681527209133' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-15 02:55:06,683 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=104a1f3a98692b3503aef02f1d865942, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:55:06,683 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-15 02:55:06,683 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34681] ipc.CallRunner(144): callId: 31 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41156 deadline: 1681527366679, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=104a1f3a98692b3503aef02f1d865942, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:06,683 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-15 02:55:06,684 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:06,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:06,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:55:06,684 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:06,684 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-15 02:55:06,685 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:06,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:55:06,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:06,685 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:06,685 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-15 02:55:06,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-15 02:55:06,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:55:06,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:06,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:55:06,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:55:06,688 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:06,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:55:06,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:06,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:55:06,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:55:06,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:06,697 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-15 02:55:06,697 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:06,697 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:55:06,697 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:55:06,697 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:06,697 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:06,697 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:06,697 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:06,697 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-15 02:55:06,697 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-15 02:55:06,697 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-15 02:55:06,697 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-15 02:55:06,698 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-15 02:55:06,698 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-15 02:55:06,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-15 02:55:06,699 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=62 (bloomFilter=true), to=hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/104a1f3a98692b3503aef02f1d865942/.tmp/B/356dd4a5b0544433b64c47453bd94424 2023-04-15 02:55:06,700 DEBUG [Thread-2181] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-15 02:55:06,700 DEBUG [Thread-2181] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-15 02:55:06,701 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-15 02:55:06,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-15 02:55:06,734 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=62 (bloomFilter=true), to=hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/104a1f3a98692b3503aef02f1d865942/.tmp/C/2a0f88f61e6c452bbd5e70ca0b1ac888 2023-04-15 02:55:06,742 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/104a1f3a98692b3503aef02f1d865942/.tmp/A/4872b863c3ac49819160a8ded98bed42 as hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/104a1f3a98692b3503aef02f1d865942/A/4872b863c3ac49819160a8ded98bed42 2023-04-15 02:55:06,746 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/104a1f3a98692b3503aef02f1d865942/A/4872b863c3ac49819160a8ded98bed42, entries=200, sequenceid=62, filesize=31.9 K 2023-04-15 02:55:06,747 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/104a1f3a98692b3503aef02f1d865942/.tmp/B/356dd4a5b0544433b64c47453bd94424 as hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/104a1f3a98692b3503aef02f1d865942/B/356dd4a5b0544433b64c47453bd94424 2023-04-15 02:55:06,750 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/104a1f3a98692b3503aef02f1d865942/B/356dd4a5b0544433b64c47453bd94424, entries=150, sequenceid=62, filesize=11.6 K 2023-04-15 02:55:06,752 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/104a1f3a98692b3503aef02f1d865942/.tmp/C/2a0f88f61e6c452bbd5e70ca0b1ac888 as hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/104a1f3a98692b3503aef02f1d865942/C/2a0f88f61e6c452bbd5e70ca0b1ac888 2023-04-15 02:55:06,756 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/104a1f3a98692b3503aef02f1d865942/C/2a0f88f61e6c452bbd5e70ca0b1ac888, entries=150, sequenceid=62, filesize=11.6 K 2023-04-15 02:55:06,757 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~60.38 KB/61830, heapSize ~180 KB/184320, currentSize=114.05 KB/116790 for 104a1f3a98692b3503aef02f1d865942 in 127ms, sequenceid=62, compaction requested=true 2023-04-15 02:55:06,757 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 104a1f3a98692b3503aef02f1d865942: 2023-04-15 02:55:06,757 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-15 02:55:06,757 DEBUG [RS:0;jenkins-hbase20:34681-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-15 02:55:06,757 DEBUG [RS:0;jenkins-hbase20:34681-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-15 02:55:06,757 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-15 02:55:06,757 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-15 02:55:06,759 DEBUG [RS:0;jenkins-hbase20:34681-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 77230 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-15 02:55:06,760 DEBUG [RS:0;jenkins-hbase20:34681-shortCompactions-0] regionserver.HStore(1912): 104a1f3a98692b3503aef02f1d865942/A is initiating minor compaction (all files) 2023-04-15 02:55:06,760 DEBUG [RS:0;jenkins-hbase20:34681-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 33314 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-15 02:55:06,760 DEBUG [RS:0;jenkins-hbase20:34681-longCompactions-0] regionserver.HStore(1912): 104a1f3a98692b3503aef02f1d865942/B is initiating minor compaction (all files) 2023-04-15 02:55:06,760 INFO [RS:0;jenkins-hbase20:34681-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 104a1f3a98692b3503aef02f1d865942/A in TestAcidGuarantees,,1681527301940.104a1f3a98692b3503aef02f1d865942. 2023-04-15 02:55:06,760 INFO [RS:0;jenkins-hbase20:34681-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/104a1f3a98692b3503aef02f1d865942/A/032ee30bdfac405b91c9b37d38512266, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/104a1f3a98692b3503aef02f1d865942/A/3cbb598a68f74c48bb9978ac9f332692, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/104a1f3a98692b3503aef02f1d865942/A/4872b863c3ac49819160a8ded98bed42] into tmpdir=hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/104a1f3a98692b3503aef02f1d865942/.tmp, totalSize=75.4 K 2023-04-15 02:55:06,760 INFO [RS:0;jenkins-hbase20:34681-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 104a1f3a98692b3503aef02f1d865942/B in TestAcidGuarantees,,1681527301940.104a1f3a98692b3503aef02f1d865942. 2023-04-15 02:55:06,760 INFO [RS:0;jenkins-hbase20:34681-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/104a1f3a98692b3503aef02f1d865942/B/697475e420914a2d85b9414f5bdfcb6e, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/104a1f3a98692b3503aef02f1d865942/B/ae3bbfafcdc747f996d9c62a7b80d13f, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/104a1f3a98692b3503aef02f1d865942/B/356dd4a5b0544433b64c47453bd94424] into tmpdir=hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/104a1f3a98692b3503aef02f1d865942/.tmp, totalSize=32.5 K 2023-04-15 02:55:06,761 DEBUG [RS:0;jenkins-hbase20:34681-shortCompactions-0] compactions.Compactor(207): Compacting 032ee30bdfac405b91c9b37d38512266, keycount=100, bloomtype=ROW, size=18.4 K, encoding=NONE, compression=NONE, seqNum=16, earliestPutTs=1681527305130 2023-04-15 02:55:06,761 DEBUG [RS:0;jenkins-hbase20:34681-longCompactions-0] compactions.Compactor(207): Compacting 697475e420914a2d85b9414f5bdfcb6e, keycount=100, bloomtype=ROW, size=9.3 K, encoding=NONE, compression=NONE, seqNum=16, earliestPutTs=1681527305130 2023-04-15 02:55:06,761 DEBUG [RS:0;jenkins-hbase20:34681-shortCompactions-0] compactions.Compactor(207): Compacting 3cbb598a68f74c48bb9978ac9f332692, keycount=150, bloomtype=ROW, size=25.1 K, encoding=NONE, compression=NONE, seqNum=50, earliestPutTs=1681527305178 2023-04-15 02:55:06,761 DEBUG [RS:0;jenkins-hbase20:34681-longCompactions-0] compactions.Compactor(207): Compacting ae3bbfafcdc747f996d9c62a7b80d13f, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=50, earliestPutTs=1681527305178 2023-04-15 02:55:06,762 DEBUG [RS:0;jenkins-hbase20:34681-longCompactions-0] compactions.Compactor(207): Compacting 356dd4a5b0544433b64c47453bd94424, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=62, earliestPutTs=1681527306309 2023-04-15 02:55:06,762 DEBUG [RS:0;jenkins-hbase20:34681-shortCompactions-0] compactions.Compactor(207): Compacting 4872b863c3ac49819160a8ded98bed42, keycount=200, bloomtype=ROW, size=31.9 K, encoding=NONE, compression=NONE, seqNum=62, earliestPutTs=1681527306301 2023-04-15 02:55:06,771 INFO [RS:0;jenkins-hbase20:34681-longCompactions-0] throttle.PressureAwareThroughputController(145): 104a1f3a98692b3503aef02f1d865942#B#compaction#284 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-15 02:55:06,785 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 104a1f3a98692b3503aef02f1d865942 3/3 column families, dataSize=120.76 KB heapSize=360.05 KB 2023-04-15 02:55:06,785 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 104a1f3a98692b3503aef02f1d865942, store=A 2023-04-15 02:55:06,785 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-15 02:55:06,785 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 104a1f3a98692b3503aef02f1d865942, store=B 2023-04-15 02:55:06,785 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-15 02:55:06,785 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 104a1f3a98692b3503aef02f1d865942, store=C 2023-04-15 02:55:06,785 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-15 02:55:06,785 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34681] regionserver.HRegion(9158): Flush requested on 104a1f3a98692b3503aef02f1d865942 2023-04-15 02:55:06,787 INFO [RS:0;jenkins-hbase20:34681-shortCompactions-0] throttle.PressureAwareThroughputController(145): 104a1f3a98692b3503aef02f1d865942#A#compaction#285 average throughput is 2.19 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-15 02:55:06,800 DEBUG [Thread-2181] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-15 02:55:06,800 DEBUG [RS:0;jenkins-hbase20:34681-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/104a1f3a98692b3503aef02f1d865942/.tmp/B/a62cf6cb551c4811ad937d6d3921e413 as hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/104a1f3a98692b3503aef02f1d865942/B/a62cf6cb551c4811ad937d6d3921e413 2023-04-15 02:55:06,800 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-15 02:55:06,802 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-15 02:55:06,805 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-15 02:55:06,805 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-15 02:55:06,805 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-15 02:55:06,805 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-15 02:55:06,805 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-15 02:55:06,805 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-15 02:55:06,806 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=104a1f3a98692b3503aef02f1d865942, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:55:06,806 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:06,806 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:55:06,806 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34681] ipc.CallRunner(144): callId: 36 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41180 deadline: 1681527366795, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=104a1f3a98692b3503aef02f1d865942, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:06,807 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-15 02:55:06,807 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:06,807 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-15 02:55:06,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-15 02:55:06,807 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:06,807 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-15 02:55:06,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:55:06,807 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:06,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-15 02:55:06,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:55:06,808 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-15 02:55:06,810 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-15 02:55:06,810 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-15 02:55:06,810 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-15 02:55:06,810 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-15 02:55:06,810 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-15 02:55:06,811 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=104a1f3a98692b3503aef02f1d865942, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:55:06,812 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34681] ipc.CallRunner(144): callId: 31 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41164 deadline: 1681527366802, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=104a1f3a98692b3503aef02f1d865942, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:06,812 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=104a1f3a98692b3503aef02f1d865942, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:55:06,812 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34681] ipc.CallRunner(144): callId: 35 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41156 deadline: 1681527366804, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=104a1f3a98692b3503aef02f1d865942, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:06,813 INFO [RS:0;jenkins-hbase20:34681-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 104a1f3a98692b3503aef02f1d865942/B of 104a1f3a98692b3503aef02f1d865942 into a62cf6cb551c4811ad937d6d3921e413(size=11.7 K), total size for store is 11.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-15 02:55:06,813 DEBUG [RS:0;jenkins-hbase20:34681-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 104a1f3a98692b3503aef02f1d865942: 2023-04-15 02:55:06,813 INFO [RS:0;jenkins-hbase20:34681-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681527301940.104a1f3a98692b3503aef02f1d865942., storeName=104a1f3a98692b3503aef02f1d865942/B, priority=13, startTime=1681527306757; duration=0sec 2023-04-15 02:55:06,813 DEBUG [RS:0;jenkins-hbase20:34681-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-15 02:55:06,813 DEBUG [RS:0;jenkins-hbase20:34681-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-15 02:55:06,814 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=104a1f3a98692b3503aef02f1d865942, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:55:06,814 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34681] ipc.CallRunner(144): callId: 29 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41118 deadline: 1681527366806, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=104a1f3a98692b3503aef02f1d865942, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:06,815 DEBUG [RS:0;jenkins-hbase20:34681-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 33314 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-15 02:55:06,815 DEBUG [RS:0;jenkins-hbase20:34681-longCompactions-0] regionserver.HStore(1912): 104a1f3a98692b3503aef02f1d865942/C is initiating minor compaction (all files) 2023-04-15 02:55:06,815 INFO [RS:0;jenkins-hbase20:34681-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 104a1f3a98692b3503aef02f1d865942/C in TestAcidGuarantees,,1681527301940.104a1f3a98692b3503aef02f1d865942. 2023-04-15 02:55:06,815 INFO [RS:0;jenkins-hbase20:34681-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/104a1f3a98692b3503aef02f1d865942/C/c48c0a175cd04484a3922fce0950b506, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/104a1f3a98692b3503aef02f1d865942/C/93e4377e68b9479f839f5be916fb3d7f, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/104a1f3a98692b3503aef02f1d865942/C/2a0f88f61e6c452bbd5e70ca0b1ac888] into tmpdir=hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/104a1f3a98692b3503aef02f1d865942/.tmp, totalSize=32.5 K 2023-04-15 02:55:06,816 DEBUG [RS:0;jenkins-hbase20:34681-longCompactions-0] compactions.Compactor(207): Compacting c48c0a175cd04484a3922fce0950b506, keycount=100, bloomtype=ROW, size=9.3 K, encoding=NONE, compression=NONE, seqNum=16, earliestPutTs=1681527305130 2023-04-15 02:55:06,816 DEBUG [RS:0;jenkins-hbase20:34681-longCompactions-0] compactions.Compactor(207): Compacting 93e4377e68b9479f839f5be916fb3d7f, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=50, earliestPutTs=1681527305178 2023-04-15 02:55:06,816 DEBUG [RS:0;jenkins-hbase20:34681-longCompactions-0] compactions.Compactor(207): Compacting 2a0f88f61e6c452bbd5e70ca0b1ac888, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=62, earliestPutTs=1681527306309 2023-04-15 02:55:06,822 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-15 02:55:06,822 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-15 02:55:06,822 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681527301940.104a1f3a98692b3503aef02f1d865942. 2023-04-15 02:55:06,822 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681527301940.104a1f3a98692b3503aef02f1d865942. started... 2023-04-15 02:55:06,822 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681527301940.104a1f3a98692b3503aef02f1d865942. as already flushing 2023-04-15 02:55:06,822 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681527301940.104a1f3a98692b3503aef02f1d865942. 2023-04-15 02:55:06,822 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-15 02:55:06,822 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-15 02:55:06,822 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:06,826 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-15 02:55:06,826 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34681,1681527209133' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-15 02:55:06,827 INFO [RS:0;jenkins-hbase20:34681-longCompactions-0] throttle.PressureAwareThroughputController(145): 104a1f3a98692b3503aef02f1d865942#C#compaction#287 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-15 02:55:06,827 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:06,827 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:06,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:06,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:55:06,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:55:06,828 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:06,828 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-15 02:55:06,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:55:06,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:55:06,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:55:06,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:06,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:55:06,841 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34681,1681527209133' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-15 02:55:06,841 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@c2d0424[Count = 0] remaining members to acquire global barrier 2023-04-15 02:55:06,841 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-15 02:55:06,841 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:06,842 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:06,842 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:06,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:06,842 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-15 02:55:06,842 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-15 02:55:06,842 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34681,1681527209133' in zk 2023-04-15 02:55:06,842 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023041542b597abb7644266aed00baa3821db15 to hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023041542b597abb7644266aed00baa3821db15 2023-04-15 02:55:06,843 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-15 02:55:06,843 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-15 02:55:06,843 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-15 02:55:06,843 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:06,844 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34681,1681527209133' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-15 02:55:06,844 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-15 02:55:06,844 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-15 02:55:06,844 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-15 02:55:06,844 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-15 02:55:06,844 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-15 02:55:06,844 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-15 02:55:06,845 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:06,845 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-15 02:55:06,845 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:06,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:06,846 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:06,846 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-15 02:55:06,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-15 02:55:06,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:06,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:55:06,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:55:06,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:06,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:55:06,847 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:06,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:55:06,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:55:06,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:55:06,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:06,849 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:06,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:55:06,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:55:06,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:06,851 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-15 02:55:06,851 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-15 02:55:06,851 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-15 02:55:06,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-15 02:55:06,852 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-15 02:55:06,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-15 02:55:06,852 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:06,852 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:55:06,852 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:55:06,852 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:06,852 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:06,852 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:06,852 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:06,853 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-15 02:55:06,853 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-15 02:55:06,853 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-15 02:55:06,853 DEBUG [Thread-2181] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-15 02:55:06,853 DEBUG [Thread-2181] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-15 02:55:06,907 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=104a1f3a98692b3503aef02f1d865942, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:55:06,907 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34681] ipc.CallRunner(144): callId: 38 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41180 deadline: 1681527366907, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=104a1f3a98692b3503aef02f1d865942, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:06,914 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=104a1f3a98692b3503aef02f1d865942, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:55:06,914 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34681] ipc.CallRunner(144): callId: 33 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41164 deadline: 1681527366913, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=104a1f3a98692b3503aef02f1d865942, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:06,920 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=104a1f3a98692b3503aef02f1d865942, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:55:06,920 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34681] ipc.CallRunner(144): callId: 37 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41156 deadline: 1681527366915, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=104a1f3a98692b3503aef02f1d865942, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:06,920 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=104a1f3a98692b3503aef02f1d865942, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:55:06,920 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34681] ipc.CallRunner(144): callId: 31 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41118 deadline: 1681527366915, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=104a1f3a98692b3503aef02f1d865942, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:06,953 DEBUG [Thread-2181] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-15 02:55:06,954 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-15 02:55:06,965 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-15 02:55:06,972 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-15 02:55:06,972 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-15 02:55:06,972 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-15 02:55:06,972 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-15 02:55:06,972 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-15 02:55:06,972 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-15 02:55:06,973 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:06,973 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:55:06,974 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-15 02:55:06,974 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:06,974 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-15 02:55:06,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-15 02:55:06,974 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:06,974 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-15 02:55:06,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:55:06,974 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:06,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-15 02:55:06,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:55:06,975 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-15 02:55:06,976 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-15 02:55:06,976 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-15 02:55:06,976 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-15 02:55:06,976 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-15 02:55:06,976 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-15 02:55:06,978 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-15 02:55:06,978 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-15 02:55:06,978 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681527301940.104a1f3a98692b3503aef02f1d865942. 2023-04-15 02:55:06,978 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681527301940.104a1f3a98692b3503aef02f1d865942. started... 2023-04-15 02:55:06,978 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681527301940.104a1f3a98692b3503aef02f1d865942. as already flushing 2023-04-15 02:55:06,978 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681527301940.104a1f3a98692b3503aef02f1d865942. 2023-04-15 02:55:06,978 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-15 02:55:06,978 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-15 02:55:06,978 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:06,978 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-15 02:55:06,978 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34681,1681527209133' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-15 02:55:06,981 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:06,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:06,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:55:06,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:55:06,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:55:06,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:55:06,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:55:06,983 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:06,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:06,983 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:06,983 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-15 02:55:06,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:55:06,983 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34681,1681527209133' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-15 02:55:06,983 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1f33d5e6[Count = 0] remaining members to acquire global barrier 2023-04-15 02:55:06,983 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-15 02:55:06,983 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:06,990 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:06,990 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:06,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:06,990 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-15 02:55:06,990 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-15 02:55:06,990 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34681,1681527209133' in zk 2023-04-15 02:55:06,990 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:06,990 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-15 02:55:06,991 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-15 02:55:06,991 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-15 02:55:06,991 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-15 02:55:06,991 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:06,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:06,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:55:06,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:55:06,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:55:06,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:55:06,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:55:06,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:06,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:55:06,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:55:06,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:06,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34681,1681527209133': 2023-04-15 02:55:06,997 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34681,1681527209133' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-15 02:55:06,997 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-15 02:55:06,997 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-15 02:55:06,997 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-15 02:55:06,997 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-15 02:55:06,997 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-15 02:55:07,005 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:07,005 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:07,005 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-15 02:55:07,006 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:07,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:07,006 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-15 02:55:07,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-15 02:55:07,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:07,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:55:07,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:07,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:55:07,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:55:07,007 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:07,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:55:07,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:55:07,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:55:07,008 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:07,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:07,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:55:07,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:55:07,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:07,010 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-15 02:55:07,010 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:07,010 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-15 02:55:07,010 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-15 02:55:07,010 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-15 02:55:07,010 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-15 02:55:07,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-15 02:55:07,010 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:55:07,010 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-15 02:55:07,011 DEBUG [Thread-2181] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-15 02:55:07,011 DEBUG [Thread-2181] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-15 02:55:07,011 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:55:07,011 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:07,011 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:07,011 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:07,011 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:07,011 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-15 02:55:07,011 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-15 02:55:07,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-15 02:55:07,111 DEBUG [Thread-2181] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-15 02:55:07,111 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-15 02:55:07,113 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-15 02:55:07,115 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-15 02:55:07,115 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=104a1f3a98692b3503aef02f1d865942, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:55:07,115 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-15 02:55:07,115 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34681] ipc.CallRunner(144): callId: 40 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41180 deadline: 1681527367108, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=104a1f3a98692b3503aef02f1d865942, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:07,115 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-15 02:55:07,116 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-15 02:55:07,116 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-15 02:55:07,116 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-15 02:55:07,116 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:07,116 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:55:07,117 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-15 02:55:07,117 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:07,117 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-15 02:55:07,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-15 02:55:07,117 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:07,117 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-15 02:55:07,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:55:07,117 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:07,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-15 02:55:07,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:55:07,118 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-15 02:55:07,118 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-15 02:55:07,118 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-15 02:55:07,118 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-15 02:55:07,118 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-15 02:55:07,118 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-15 02:55:07,123 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-15 02:55:07,123 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-15 02:55:07,123 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681527301940.104a1f3a98692b3503aef02f1d865942. 2023-04-15 02:55:07,123 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681527301940.104a1f3a98692b3503aef02f1d865942. started... 2023-04-15 02:55:07,123 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681527301940.104a1f3a98692b3503aef02f1d865942. as already flushing 2023-04-15 02:55:07,123 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681527301940.104a1f3a98692b3503aef02f1d865942. 2023-04-15 02:55:07,123 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-15 02:55:07,123 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-15 02:55:07,123 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:07,123 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-15 02:55:07,123 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34681,1681527209133' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-15 02:55:07,124 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=104a1f3a98692b3503aef02f1d865942, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:55:07,124 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34681] ipc.CallRunner(144): callId: 35 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41164 deadline: 1681527367116, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=104a1f3a98692b3503aef02f1d865942, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:07,126 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=104a1f3a98692b3503aef02f1d865942, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:55:07,126 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34681] ipc.CallRunner(144): callId: 39 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41156 deadline: 1681527367121, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=104a1f3a98692b3503aef02f1d865942, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:07,126 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=104a1f3a98692b3503aef02f1d865942, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:55:07,126 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34681] ipc.CallRunner(144): callId: 33 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41118 deadline: 1681527367121, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=104a1f3a98692b3503aef02f1d865942, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:07,130 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:07,130 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:07,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:07,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:55:07,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:55:07,130 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:07,130 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-15 02:55:07,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:55:07,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:55:07,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:55:07,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:07,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:55:07,131 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34681,1681527209133' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-15 02:55:07,131 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@19fa9ad3[Count = 0] remaining members to acquire global barrier 2023-04-15 02:55:07,131 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-15 02:55:07,131 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:07,132 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:07,132 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:07,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:07,132 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-15 02:55:07,132 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-15 02:55:07,132 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34681,1681527209133' in zk 2023-04-15 02:55:07,132 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:07,132 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-15 02:55:07,133 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:07,133 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-15 02:55:07,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:07,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:55:07,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:55:07,133 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-15 02:55:07,133 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-15 02:55:07,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:55:07,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:55:07,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:55:07,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:07,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:55:07,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:55:07,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:07,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34681,1681527209133': 2023-04-15 02:55:07,135 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34681,1681527209133' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-15 02:55:07,135 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-15 02:55:07,135 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-15 02:55:07,135 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-15 02:55:07,135 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-15 02:55:07,135 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-15 02:55:07,136 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:07,136 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:07,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:07,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:55:07,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:55:07,136 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:07,136 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-15 02:55:07,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:07,136 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-15 02:55:07,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-15 02:55:07,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:07,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:55:07,137 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:07,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:55:07,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:55:07,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:55:07,138 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:07,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:07,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:55:07,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:55:07,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:07,140 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:07,140 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-15 02:55:07,140 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:55:07,140 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-15 02:55:07,140 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-15 02:55:07,140 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:55:07,140 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-15 02:55:07,140 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-15 02:55:07,140 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:07,140 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-15 02:55:07,140 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-15 02:55:07,140 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:07,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-15 02:55:07,140 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:07,140 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:07,140 DEBUG [Thread-2181] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-15 02:55:07,140 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-15 02:55:07,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-15 02:55:07,140 DEBUG [Thread-2181] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-15 02:55:07,241 DEBUG [Thread-2181] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-15 02:55:07,241 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-15 02:55:07,242 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-15 02:55:07,243 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-15 02:55:07,243 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-15 02:55:07,243 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-15 02:55:07,243 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-15 02:55:07,243 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-15 02:55:07,243 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-15 02:55:07,244 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:07,244 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:55:07,244 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-15 02:55:07,244 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:07,244 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-15 02:55:07,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-15 02:55:07,245 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:07,245 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-15 02:55:07,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:55:07,245 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:07,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-15 02:55:07,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:55:07,245 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-15 02:55:07,245 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-15 02:55:07,245 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-15 02:55:07,245 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-15 02:55:07,246 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-15 02:55:07,246 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-15 02:55:07,246 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-15 02:55:07,246 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-15 02:55:07,246 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681527301940.104a1f3a98692b3503aef02f1d865942. 2023-04-15 02:55:07,246 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681527301940.104a1f3a98692b3503aef02f1d865942. started... 2023-04-15 02:55:07,246 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681527301940.104a1f3a98692b3503aef02f1d865942. as already flushing 2023-04-15 02:55:07,246 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681527301940.104a1f3a98692b3503aef02f1d865942. 2023-04-15 02:55:07,246 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-15 02:55:07,246 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-15 02:55:07,246 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:07,246 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-15 02:55:07,246 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34681,1681527209133' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-15 02:55:07,247 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:07,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:07,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:55:07,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:55:07,247 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:07,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:55:07,247 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:07,247 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-15 02:55:07,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:55:07,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:55:07,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:07,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:55:07,249 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34681,1681527209133' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-15 02:55:07,249 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7fa0b691[Count = 0] remaining members to acquire global barrier 2023-04-15 02:55:07,250 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-15 02:55:07,250 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:07,251 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:07,251 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:07,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:07,251 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-15 02:55:07,251 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-15 02:55:07,251 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34681,1681527209133' in zk 2023-04-15 02:55:07,251 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:07,251 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-15 02:55:07,252 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:07,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:07,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:55:07,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:55:07,252 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-15 02:55:07,252 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-15 02:55:07,252 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-15 02:55:07,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:55:07,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:55:07,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:55:07,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:07,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:55:07,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:55:07,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:07,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34681,1681527209133': 2023-04-15 02:55:07,253 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34681,1681527209133' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-15 02:55:07,254 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-15 02:55:07,254 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-15 02:55:07,254 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-15 02:55:07,254 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-15 02:55:07,254 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-15 02:55:07,254 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:07,254 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:07,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:07,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:55:07,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:55:07,254 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:07,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:07,254 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-15 02:55:07,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:55:07,255 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:07,255 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-15 02:55:07,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-15 02:55:07,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:55:07,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:07,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:55:07,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:55:07,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:07,256 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:07,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:55:07,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:55:07,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:07,263 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-15 02:55:07,263 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-15 02:55:07,263 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-15 02:55:07,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-15 02:55:07,263 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:07,264 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:55:07,264 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:55:07,264 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:07,264 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:07,264 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:07,264 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:07,264 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-15 02:55:07,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-15 02:55:07,266 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-15 02:55:07,266 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-15 02:55:07,266 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-15 02:55:07,272 DEBUG [RS:0;jenkins-hbase20:34681-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/104a1f3a98692b3503aef02f1d865942/.tmp/C/355d44d68c824843ad7c1f9e8f03cfe4 as hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/104a1f3a98692b3503aef02f1d865942/C/355d44d68c824843ad7c1f9e8f03cfe4 2023-04-15 02:55:07,274 DEBUG [Thread-2181] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-15 02:55:07,274 DEBUG [Thread-2181] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-15 02:55:07,277 INFO [RS:0;jenkins-hbase20:34681-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 104a1f3a98692b3503aef02f1d865942/C of 104a1f3a98692b3503aef02f1d865942 into 355d44d68c824843ad7c1f9e8f03cfe4(size=11.7 K), total size for store is 11.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-15 02:55:07,277 DEBUG [RS:0;jenkins-hbase20:34681-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 104a1f3a98692b3503aef02f1d865942: 2023-04-15 02:55:07,277 INFO [RS:0;jenkins-hbase20:34681-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681527301940.104a1f3a98692b3503aef02f1d865942., storeName=104a1f3a98692b3503aef02f1d865942/C, priority=13, startTime=1681527306757; duration=0sec 2023-04-15 02:55:07,277 DEBUG [RS:0;jenkins-hbase20:34681-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-15 02:55:07,278 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=83, memsize=40.3 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/104a1f3a98692b3503aef02f1d865942/.tmp/A/991d654bc57544848c53ef515055bdc8 2023-04-15 02:55:07,293 DEBUG [RS:0;jenkins-hbase20:34681-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/104a1f3a98692b3503aef02f1d865942/.tmp/A/2baf48876e504d309b0eb53ae2597104 as hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/104a1f3a98692b3503aef02f1d865942/A/2baf48876e504d309b0eb53ae2597104 2023-04-15 02:55:07,300 INFO [RS:0;jenkins-hbase20:34681-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 104a1f3a98692b3503aef02f1d865942/A of 104a1f3a98692b3503aef02f1d865942 into 2baf48876e504d309b0eb53ae2597104(size=25.2 K), total size for store is 25.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-15 02:55:07,300 DEBUG [RS:0;jenkins-hbase20:34681-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 104a1f3a98692b3503aef02f1d865942: 2023-04-15 02:55:07,300 INFO [RS:0;jenkins-hbase20:34681-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681527301940.104a1f3a98692b3503aef02f1d865942., storeName=104a1f3a98692b3503aef02f1d865942/A, priority=13, startTime=1681527306757; duration=0sec 2023-04-15 02:55:07,300 DEBUG [RS:0;jenkins-hbase20:34681-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-15 02:55:07,323 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=104a1f3a98692b3503aef02f1d865942, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:55:07,323 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34681] ipc.CallRunner(144): callId: 20 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41166 deadline: 1681527367316, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=104a1f3a98692b3503aef02f1d865942, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:07,329 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=40.25 KB at sequenceid=83 (bloomFilter=true), to=hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/104a1f3a98692b3503aef02f1d865942/.tmp/B/7d51e0aeb60340a3b6351c7a7e06866e 2023-04-15 02:55:07,374 DEBUG [Thread-2181] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-15 02:55:07,375 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-15 02:55:07,376 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-15 02:55:07,386 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-15 02:55:07,386 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-15 02:55:07,386 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-15 02:55:07,386 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-15 02:55:07,386 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-15 02:55:07,386 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-15 02:55:07,386 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:07,387 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:55:07,387 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:07,387 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-15 02:55:07,387 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-15 02:55:07,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-15 02:55:07,389 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:07,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:55:07,389 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-15 02:55:07,389 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:07,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-15 02:55:07,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:55:07,390 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-15 02:55:07,390 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-15 02:55:07,390 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-15 02:55:07,390 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-15 02:55:07,390 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-15 02:55:07,390 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-15 02:55:07,398 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-15 02:55:07,398 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-15 02:55:07,398 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681527301940.104a1f3a98692b3503aef02f1d865942. 2023-04-15 02:55:07,398 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681527301940.104a1f3a98692b3503aef02f1d865942. started... 2023-04-15 02:55:07,398 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681527301940.104a1f3a98692b3503aef02f1d865942. as already flushing 2023-04-15 02:55:07,398 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681527301940.104a1f3a98692b3503aef02f1d865942. 2023-04-15 02:55:07,398 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-15 02:55:07,398 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-15 02:55:07,398 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:07,398 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-15 02:55:07,398 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34681,1681527209133' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-15 02:55:07,405 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:07,405 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:07,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:07,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:55:07,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:55:07,405 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:07,405 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-15 02:55:07,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:55:07,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:55:07,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:55:07,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:07,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:55:07,407 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34681,1681527209133' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-15 02:55:07,407 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1b488165[Count = 0] remaining members to acquire global barrier 2023-04-15 02:55:07,407 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-15 02:55:07,407 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:07,415 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:07,416 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:07,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:07,416 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-15 02:55:07,416 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-15 02:55:07,416 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34681,1681527209133' in zk 2023-04-15 02:55:07,416 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:07,416 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-15 02:55:07,422 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=104a1f3a98692b3503aef02f1d865942, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:55:07,422 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34681] ipc.CallRunner(144): callId: 42 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41180 deadline: 1681527367417, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=104a1f3a98692b3503aef02f1d865942, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:07,423 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-15 02:55:07,423 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-15 02:55:07,423 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-15 02:55:07,423 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:07,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:07,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:55:07,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:55:07,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:55:07,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:55:07,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:55:07,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:07,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:55:07,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:55:07,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:07,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34681,1681527209133': 2023-04-15 02:55:07,427 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34681,1681527209133' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-15 02:55:07,427 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-15 02:55:07,427 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-15 02:55:07,427 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-15 02:55:07,427 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-15 02:55:07,427 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-15 02:55:07,431 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=104a1f3a98692b3503aef02f1d865942, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:55:07,431 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34681] ipc.CallRunner(144): callId: 37 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41164 deadline: 1681527367427, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=104a1f3a98692b3503aef02f1d865942, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:07,431 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:07,431 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-15 02:55:07,431 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:07,431 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:07,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:07,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:55:07,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:55:07,431 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=104a1f3a98692b3503aef02f1d865942, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:55:07,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:07,432 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34681] ipc.CallRunner(144): callId: 35 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41118 deadline: 1681527367427, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=104a1f3a98692b3503aef02f1d865942, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:07,433 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:07,433 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=104a1f3a98692b3503aef02f1d865942, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:55:07,432 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-15 02:55:07,434 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34681] ipc.CallRunner(144): callId: 41 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41156 deadline: 1681527367429, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=104a1f3a98692b3503aef02f1d865942, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:07,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:55:07,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-15 02:55:07,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:55:07,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:07,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:55:07,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:55:07,435 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:07,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:07,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:55:07,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:55:07,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:07,441 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:07,441 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:55:07,441 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:55:07,441 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-15 02:55:07,441 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-15 02:55:07,441 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:07,441 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:07,441 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:07,441 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:07,441 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-15 02:55:07,441 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-15 02:55:07,442 DEBUG [Thread-2181] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-15 02:55:07,442 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-15 02:55:07,442 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-15 02:55:07,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-15 02:55:07,442 DEBUG [Thread-2181] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-15 02:55:07,442 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-15 02:55:07,442 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-15 02:55:07,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-15 02:55:07,542 DEBUG [Thread-2181] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-15 02:55:07,543 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-15 02:55:07,544 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-15 02:55:07,545 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-15 02:55:07,545 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-15 02:55:07,545 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-15 02:55:07,545 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-15 02:55:07,545 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-15 02:55:07,545 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-15 02:55:07,545 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:07,545 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:55:07,546 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-15 02:55:07,546 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-15 02:55:07,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-15 02:55:07,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:55:07,547 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:07,547 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:07,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-15 02:55:07,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:55:07,547 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-15 02:55:07,548 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:07,548 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-15 02:55:07,548 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-15 02:55:07,548 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-15 02:55:07,548 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-15 02:55:07,548 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-15 02:55:07,548 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-15 02:55:07,548 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-15 02:55:07,548 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-15 02:55:07,548 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681527301940.104a1f3a98692b3503aef02f1d865942. 2023-04-15 02:55:07,548 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681527301940.104a1f3a98692b3503aef02f1d865942. started... 2023-04-15 02:55:07,548 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681527301940.104a1f3a98692b3503aef02f1d865942. as already flushing 2023-04-15 02:55:07,549 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681527301940.104a1f3a98692b3503aef02f1d865942. 2023-04-15 02:55:07,549 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-15 02:55:07,549 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-15 02:55:07,549 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:07,549 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-15 02:55:07,549 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34681,1681527209133' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-15 02:55:07,549 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:07,549 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:07,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:07,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:55:07,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:55:07,550 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:07,550 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-15 02:55:07,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:55:07,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:55:07,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:55:07,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:07,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:55:07,550 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34681,1681527209133' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-15 02:55:07,551 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1a55fb0f[Count = 0] remaining members to acquire global barrier 2023-04-15 02:55:07,551 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-15 02:55:07,551 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:07,551 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:07,551 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:07,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:07,551 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-15 02:55:07,551 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:07,551 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-15 02:55:07,551 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-15 02:55:07,551 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34681,1681527209133' in zk 2023-04-15 02:55:07,552 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:07,552 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-15 02:55:07,552 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-15 02:55:07,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:07,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:55:07,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:55:07,552 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-15 02:55:07,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:55:07,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:55:07,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:55:07,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:07,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:55:07,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:55:07,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:07,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34681,1681527209133': 2023-04-15 02:55:07,554 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34681,1681527209133' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-15 02:55:07,554 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-15 02:55:07,554 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-15 02:55:07,554 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-15 02:55:07,554 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-15 02:55:07,554 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-15 02:55:07,555 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:07,555 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-15 02:55:07,555 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:07,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:07,555 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:07,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:07,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:55:07,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:55:07,555 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-15 02:55:07,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-15 02:55:07,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:55:07,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:07,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:55:07,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:55:07,556 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:07,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:55:07,557 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:07,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:07,558 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:07,558 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:55:07,558 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:55:07,558 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-15 02:55:07,558 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-15 02:55:07,558 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:07,558 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-15 02:55:07,558 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-15 02:55:07,558 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-15 02:55:07,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:55:07,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-15 02:55:07,558 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-15 02:55:07,558 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:07,558 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:07,558 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:07,558 DEBUG [Thread-2181] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-15 02:55:07,558 DEBUG [Thread-2181] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-15 02:55:07,559 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-15 02:55:07,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-15 02:55:07,560 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-15 02:55:07,659 DEBUG [Thread-2181] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-15 02:55:07,659 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-15 02:55:07,660 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-15 02:55:07,661 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-15 02:55:07,661 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-15 02:55:07,661 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-15 02:55:07,661 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-15 02:55:07,661 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-15 02:55:07,661 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-15 02:55:07,661 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:07,661 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:55:07,662 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-15 02:55:07,662 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:07,662 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-15 02:55:07,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-15 02:55:07,662 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:07,662 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-15 02:55:07,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:55:07,663 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:07,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-15 02:55:07,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:55:07,663 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-15 02:55:07,663 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-15 02:55:07,663 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-15 02:55:07,663 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-15 02:55:07,663 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-15 02:55:07,663 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-15 02:55:07,663 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-15 02:55:07,663 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-15 02:55:07,663 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681527301940.104a1f3a98692b3503aef02f1d865942. 2023-04-15 02:55:07,663 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681527301940.104a1f3a98692b3503aef02f1d865942. started... 2023-04-15 02:55:07,663 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681527301940.104a1f3a98692b3503aef02f1d865942. as already flushing 2023-04-15 02:55:07,663 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681527301940.104a1f3a98692b3503aef02f1d865942. 2023-04-15 02:55:07,664 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-15 02:55:07,664 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-15 02:55:07,664 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:07,664 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-15 02:55:07,664 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34681,1681527209133' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-15 02:55:07,665 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:07,665 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:07,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:07,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:55:07,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:55:07,666 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:07,666 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-15 02:55:07,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:55:07,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:55:07,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:55:07,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:07,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:55:07,675 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34681,1681527209133' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-15 02:55:07,675 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@57b4c694[Count = 0] remaining members to acquire global barrier 2023-04-15 02:55:07,675 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-15 02:55:07,675 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:07,681 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:07,681 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:07,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:07,681 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-15 02:55:07,682 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-15 02:55:07,682 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34681,1681527209133' in zk 2023-04-15 02:55:07,682 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:07,682 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-15 02:55:07,682 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:07,682 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-15 02:55:07,682 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-15 02:55:07,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:07,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:55:07,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:55:07,682 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-15 02:55:07,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:55:07,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:55:07,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:55:07,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:07,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:55:07,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:55:07,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:07,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34681,1681527209133': 2023-04-15 02:55:07,684 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34681,1681527209133' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-15 02:55:07,684 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-15 02:55:07,684 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-15 02:55:07,684 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-15 02:55:07,685 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-15 02:55:07,685 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-15 02:55:07,685 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:07,685 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:07,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:07,685 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:07,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:55:07,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:55:07,685 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-15 02:55:07,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:07,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:55:07,686 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-15 02:55:07,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-15 02:55:07,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:55:07,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:07,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:55:07,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:55:07,687 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:07,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:07,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:55:07,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:55:07,688 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:07,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:07,690 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-15 02:55:07,690 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-15 02:55:07,690 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-15 02:55:07,690 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:07,690 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-15 02:55:07,690 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-15 02:55:07,690 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-15 02:55:07,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-15 02:55:07,690 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:55:07,690 DEBUG [Thread-2181] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-15 02:55:07,690 DEBUG [Thread-2181] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-15 02:55:07,690 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:55:07,690 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:07,690 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:07,690 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:07,690 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:07,691 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-15 02:55:07,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-15 02:55:07,771 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=40.25 KB at sequenceid=83 (bloomFilter=true), to=hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/104a1f3a98692b3503aef02f1d865942/.tmp/C/520d53b28c344c838f7c4769d039de8c 2023-04-15 02:55:07,775 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/104a1f3a98692b3503aef02f1d865942/.tmp/A/991d654bc57544848c53ef515055bdc8 as hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/104a1f3a98692b3503aef02f1d865942/A/991d654bc57544848c53ef515055bdc8 2023-04-15 02:55:07,782 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/104a1f3a98692b3503aef02f1d865942/A/991d654bc57544848c53ef515055bdc8, entries=200, sequenceid=83, filesize=31.9 K 2023-04-15 02:55:07,786 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/104a1f3a98692b3503aef02f1d865942/.tmp/B/7d51e0aeb60340a3b6351c7a7e06866e as hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/104a1f3a98692b3503aef02f1d865942/B/7d51e0aeb60340a3b6351c7a7e06866e 2023-04-15 02:55:07,790 DEBUG [Thread-2181] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-15 02:55:07,791 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-15 02:55:07,793 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-15 02:55:07,803 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/104a1f3a98692b3503aef02f1d865942/B/7d51e0aeb60340a3b6351c7a7e06866e, entries=150, sequenceid=83, filesize=11.6 K 2023-04-15 02:55:07,804 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/104a1f3a98692b3503aef02f1d865942/.tmp/C/520d53b28c344c838f7c4769d039de8c as hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/104a1f3a98692b3503aef02f1d865942/C/520d53b28c344c838f7c4769d039de8c 2023-04-15 02:55:07,806 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-15 02:55:07,806 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-15 02:55:07,806 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-15 02:55:07,806 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-15 02:55:07,806 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-15 02:55:07,806 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-15 02:55:07,807 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:07,807 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:55:07,807 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:07,807 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-15 02:55:07,807 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-15 02:55:07,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-15 02:55:07,808 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:07,808 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-15 02:55:07,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:55:07,808 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:07,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-15 02:55:07,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:55:07,808 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-15 02:55:07,828 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-15 02:55:07,829 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-15 02:55:07,829 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-15 02:55:07,829 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-15 02:55:07,829 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-15 02:55:07,849 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/104a1f3a98692b3503aef02f1d865942/C/520d53b28c344c838f7c4769d039de8c, entries=150, sequenceid=83, filesize=11.6 K 2023-04-15 02:55:07,850 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~120.76 KB/123660, heapSize ~360 KB/368640, currentSize=53.67 KB/54960 for 104a1f3a98692b3503aef02f1d865942 in 1066ms, sequenceid=83, compaction requested=false 2023-04-15 02:55:07,850 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 104a1f3a98692b3503aef02f1d865942: 2023-04-15 02:55:07,866 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-15 02:55:07,866 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681527301940.104a1f3a98692b3503aef02f1d865942. 2023-04-15 02:55:07,866 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-15 02:55:07,866 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681527301940.104a1f3a98692b3503aef02f1d865942. started... 2023-04-15 02:55:07,866 INFO [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 104a1f3a98692b3503aef02f1d865942 3/3 column families, dataSize=53.67 KB heapSize=160.13 KB 2023-04-15 02:55:07,866 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 104a1f3a98692b3503aef02f1d865942, store=A 2023-04-15 02:55:07,867 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-15 02:55:07,867 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 104a1f3a98692b3503aef02f1d865942, store=B 2023-04-15 02:55:07,867 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-15 02:55:07,867 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 104a1f3a98692b3503aef02f1d865942, store=C 2023-04-15 02:55:07,867 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-15 02:55:07,930 INFO [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023041531ca4f9c67eb42958e71291b2f44727a to hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023041531ca4f9c67eb42958e71291b2f44727a 2023-04-15 02:55:07,939 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34681] regionserver.HRegion(9158): Flush requested on 104a1f3a98692b3503aef02f1d865942 2023-04-15 02:55:07,939 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681527301940.104a1f3a98692b3503aef02f1d865942. as already flushing 2023-04-15 02:55:07,953 INFO [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=97, memsize=17.9 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/104a1f3a98692b3503aef02f1d865942/.tmp/A/c91da5073e754463afa07efe85663f69 2023-04-15 02:55:07,976 INFO [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=97 (bloomFilter=true), to=hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/104a1f3a98692b3503aef02f1d865942/.tmp/B/f5c1bacbe88140468f37d7341a61ca1d 2023-04-15 02:55:07,981 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=359, cellSize=16435, totalHeapSize=37576, min timestamp=1681527307952, max timestamp=1681527307979Num uniques 859; 2023-04-15 02:55:07,981 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-2] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=16435, suffix heap size=117064,new segment heap size=37576 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=359 2023-04-15 02:55:07,981 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=359, cellSize=16435, totalHeapSize=37576, min timestamp=1681527307952, max timestamp=1681527307979Num uniques 859; 2023-04-15 02:55:07,981 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=359, cellSize=16435, totalHeapSize=37576, min timestamp=1681527307952, max timestamp=1681527307979Num uniques 859; 2023-04-15 02:55:07,981 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-7] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=16435, suffix heap size=117064,new segment heap size=37576 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=359 2023-04-15 02:55:07,981 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=16435, suffix heap size=117064,new segment heap size=37576 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=359 2023-04-15 02:55:08,010 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=104a1f3a98692b3503aef02f1d865942, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:55:08,010 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34681] ipc.CallRunner(144): callId: 44 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41118 deadline: 1681527368002, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=104a1f3a98692b3503aef02f1d865942, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:08,010 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=104a1f3a98692b3503aef02f1d865942, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:55:08,010 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34681] ipc.CallRunner(144): callId: 46 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41164 deadline: 1681527368003, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=104a1f3a98692b3503aef02f1d865942, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:08,011 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=104a1f3a98692b3503aef02f1d865942, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:55:08,011 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34681] ipc.CallRunner(144): callId: 50 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41156 deadline: 1681527368009, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=104a1f3a98692b3503aef02f1d865942, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:08,023 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=104a1f3a98692b3503aef02f1d865942, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:55:08,023 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34681] ipc.CallRunner(144): callId: 53 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41180 deadline: 1681527368010, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=104a1f3a98692b3503aef02f1d865942, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:08,116 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=104a1f3a98692b3503aef02f1d865942, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:55:08,116 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34681] ipc.CallRunner(144): callId: 46 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41118 deadline: 1681527368111, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=104a1f3a98692b3503aef02f1d865942, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:08,116 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=104a1f3a98692b3503aef02f1d865942, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:55:08,116 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34681] ipc.CallRunner(144): callId: 48 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41164 deadline: 1681527368111, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=104a1f3a98692b3503aef02f1d865942, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:08,116 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=104a1f3a98692b3503aef02f1d865942, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:55:08,117 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34681] ipc.CallRunner(144): callId: 52 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41156 deadline: 1681527368111, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=104a1f3a98692b3503aef02f1d865942, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:08,128 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=104a1f3a98692b3503aef02f1d865942, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:55:08,128 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34681] ipc.CallRunner(144): callId: 55 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41180 deadline: 1681527368124, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=104a1f3a98692b3503aef02f1d865942, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:08,321 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=104a1f3a98692b3503aef02f1d865942, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:55:08,321 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34681] ipc.CallRunner(144): callId: 48 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41118 deadline: 1681527368317, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=104a1f3a98692b3503aef02f1d865942, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:08,321 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=104a1f3a98692b3503aef02f1d865942, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:55:08,321 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34681] ipc.CallRunner(144): callId: 50 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41164 deadline: 1681527368317, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=104a1f3a98692b3503aef02f1d865942, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:08,321 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=104a1f3a98692b3503aef02f1d865942, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:55:08,321 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34681] ipc.CallRunner(144): callId: 54 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41156 deadline: 1681527368317, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=104a1f3a98692b3503aef02f1d865942, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:08,333 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=104a1f3a98692b3503aef02f1d865942, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:55:08,334 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34681] ipc.CallRunner(144): callId: 57 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41180 deadline: 1681527368329, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=104a1f3a98692b3503aef02f1d865942, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:08,396 INFO [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=97 (bloomFilter=true), to=hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/104a1f3a98692b3503aef02f1d865942/.tmp/C/ad3099311aa949b68b3948d71019ee48 2023-04-15 02:55:08,400 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/104a1f3a98692b3503aef02f1d865942/.tmp/A/c91da5073e754463afa07efe85663f69 as hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/104a1f3a98692b3503aef02f1d865942/A/c91da5073e754463afa07efe85663f69 2023-04-15 02:55:08,403 INFO [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/104a1f3a98692b3503aef02f1d865942/A/c91da5073e754463afa07efe85663f69, entries=150, sequenceid=97, filesize=25.1 K 2023-04-15 02:55:08,403 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/104a1f3a98692b3503aef02f1d865942/.tmp/B/f5c1bacbe88140468f37d7341a61ca1d as hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/104a1f3a98692b3503aef02f1d865942/B/f5c1bacbe88140468f37d7341a61ca1d 2023-04-15 02:55:08,407 INFO [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/104a1f3a98692b3503aef02f1d865942/B/f5c1bacbe88140468f37d7341a61ca1d, entries=150, sequenceid=97, filesize=11.6 K 2023-04-15 02:55:08,408 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/104a1f3a98692b3503aef02f1d865942/.tmp/C/ad3099311aa949b68b3948d71019ee48 as hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/104a1f3a98692b3503aef02f1d865942/C/ad3099311aa949b68b3948d71019ee48 2023-04-15 02:55:08,411 INFO [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/104a1f3a98692b3503aef02f1d865942/C/ad3099311aa949b68b3948d71019ee48, entries=150, sequenceid=97, filesize=11.6 K 2023-04-15 02:55:08,412 INFO [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=134.18 KB/137400 for 104a1f3a98692b3503aef02f1d865942 in 546ms, sequenceid=97, compaction requested=true 2023-04-15 02:55:08,412 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 104a1f3a98692b3503aef02f1d865942: 2023-04-15 02:55:08,412 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681527301940.104a1f3a98692b3503aef02f1d865942. 2023-04-15 02:55:08,412 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-15 02:55:08,412 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-15 02:55:08,412 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:08,412 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-15 02:55:08,412 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34681,1681527209133' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-15 02:55:08,413 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:08,413 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:08,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:08,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:55:08,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:55:08,413 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:08,413 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-15 02:55:08,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:55:08,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:55:08,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:55:08,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:08,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:55:08,414 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34681,1681527209133' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-15 02:55:08,414 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2d5ae6cd[Count = 0] remaining members to acquire global barrier 2023-04-15 02:55:08,414 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-15 02:55:08,414 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:08,415 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:08,415 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:08,415 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-15 02:55:08,416 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:08,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:08,416 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-15 02:55:08,416 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-15 02:55:08,416 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34681,1681527209133' in zk 2023-04-15 02:55:08,417 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-15 02:55:08,417 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:08,417 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-15 02:55:08,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:08,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:55:08,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:55:08,417 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-15 02:55:08,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:55:08,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:55:08,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:55:08,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:08,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:55:08,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:55:08,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:08,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34681,1681527209133': 2023-04-15 02:55:08,419 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34681,1681527209133' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-15 02:55:08,419 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-15 02:55:08,419 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-15 02:55:08,419 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-15 02:55:08,419 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-15 02:55:08,419 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-15 02:55:08,420 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:08,420 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:08,420 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:08,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:08,420 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-15 02:55:08,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:08,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:55:08,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:55:08,421 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-15 02:55:08,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-15 02:55:08,421 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:08,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:08,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:55:08,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:55:08,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:55:08,422 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:08,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:55:08,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:08,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:55:08,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:55:08,424 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-15 02:55:08,424 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:08,424 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-15 02:55:08,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-15 02:55:08,424 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-15 02:55:08,424 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-15 02:55:08,424 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:55:08,424 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:55:08,424 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:08,424 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:08,424 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:08,424 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:08,424 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-15 02:55:08,424 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-15 02:55:08,424 DEBUG [Thread-2181] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-15 02:55:08,424 DEBUG [Thread-2181] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-15 02:55:08,424 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-15 02:55:08,424 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-15 02:55:08,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-15 02:55:08,524 DEBUG [Thread-2181] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-15 02:55:08,525 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-15 02:55:08,526 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-15 02:55:08,527 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-15 02:55:08,527 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-15 02:55:08,527 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-15 02:55:08,527 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-15 02:55:08,527 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-15 02:55:08,527 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-15 02:55:08,528 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:08,528 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:55:08,528 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-15 02:55:08,528 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:08,528 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-15 02:55:08,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-15 02:55:08,528 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:08,528 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-15 02:55:08,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:55:08,529 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:08,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-15 02:55:08,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:55:08,529 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-15 02:55:08,529 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-15 02:55:08,529 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-15 02:55:08,529 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-15 02:55:08,529 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-15 02:55:08,529 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-15 02:55:08,530 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-15 02:55:08,530 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-15 02:55:08,530 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681527301940.104a1f3a98692b3503aef02f1d865942. 2023-04-15 02:55:08,530 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681527301940.104a1f3a98692b3503aef02f1d865942. started... 2023-04-15 02:55:08,530 INFO [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 104a1f3a98692b3503aef02f1d865942 3/3 column families, dataSize=134.18 KB heapSize=366.23 KB 2023-04-15 02:55:08,530 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 104a1f3a98692b3503aef02f1d865942, store=A 2023-04-15 02:55:08,530 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-15 02:55:08,530 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 104a1f3a98692b3503aef02f1d865942, store=B 2023-04-15 02:55:08,530 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-15 02:55:08,530 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 104a1f3a98692b3503aef02f1d865942, store=C 2023-04-15 02:55:08,530 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-15 02:55:08,627 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34681] regionserver.HRegion(9158): Flush requested on 104a1f3a98692b3503aef02f1d865942 2023-04-15 02:55:08,627 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681527301940.104a1f3a98692b3503aef02f1d865942. as already flushing 2023-04-15 02:55:08,641 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=104a1f3a98692b3503aef02f1d865942, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:55:08,641 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34681] ipc.CallRunner(144): callId: 59 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41180 deadline: 1681527368635, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=104a1f3a98692b3503aef02f1d865942, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:08,642 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=104a1f3a98692b3503aef02f1d865942, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:55:08,642 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34681] ipc.CallRunner(144): callId: 53 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41118 deadline: 1681527368636, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=104a1f3a98692b3503aef02f1d865942, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:08,648 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=104a1f3a98692b3503aef02f1d865942, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:55:08,648 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34681] ipc.CallRunner(144): callId: 55 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41164 deadline: 1681527368640, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=104a1f3a98692b3503aef02f1d865942, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:08,651 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=104a1f3a98692b3503aef02f1d865942, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:55:08,651 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34681] ipc.CallRunner(144): callId: 59 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41156 deadline: 1681527368641, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=104a1f3a98692b3503aef02f1d865942, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:08,748 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=104a1f3a98692b3503aef02f1d865942, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:55:08,748 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34681] ipc.CallRunner(144): callId: 55 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41118 deadline: 1681527368743, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=104a1f3a98692b3503aef02f1d865942, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:08,753 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=104a1f3a98692b3503aef02f1d865942, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:55:08,753 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34681] ipc.CallRunner(144): callId: 57 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41164 deadline: 1681527368749, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=104a1f3a98692b3503aef02f1d865942, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:08,757 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=104a1f3a98692b3503aef02f1d865942, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:55:08,757 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34681] ipc.CallRunner(144): callId: 61 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41156 deadline: 1681527368752, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=104a1f3a98692b3503aef02f1d865942, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:08,940 INFO [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230415d32d20facef14127baf84eaf7920ec6f to hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230415d32d20facef14127baf84eaf7920ec6f 2023-04-15 02:55:08,944 INFO [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=130, memsize=44.7 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/104a1f3a98692b3503aef02f1d865942/.tmp/A/93140167335e47b4970764378c84ea19 2023-04-15 02:55:08,958 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=104a1f3a98692b3503aef02f1d865942, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:55:08,958 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34681] ipc.CallRunner(144): callId: 57 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41118 deadline: 1681527368950, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=104a1f3a98692b3503aef02f1d865942, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:08,958 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=104a1f3a98692b3503aef02f1d865942, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:55:08,958 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34681] ipc.CallRunner(144): callId: 59 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41164 deadline: 1681527368954, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=104a1f3a98692b3503aef02f1d865942, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:08,966 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=104a1f3a98692b3503aef02f1d865942, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:55:08,966 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34681] ipc.CallRunner(144): callId: 63 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41156 deadline: 1681527368959, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=104a1f3a98692b3503aef02f1d865942, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:09,143 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=104a1f3a98692b3503aef02f1d865942, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:55:09,143 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34681] ipc.CallRunner(144): callId: 61 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41180 deadline: 1681527369142, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=104a1f3a98692b3503aef02f1d865942, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:09,262 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=104a1f3a98692b3503aef02f1d865942, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:55:09,262 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34681] ipc.CallRunner(144): callId: 59 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41118 deadline: 1681527369260, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=104a1f3a98692b3503aef02f1d865942, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:09,262 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=104a1f3a98692b3503aef02f1d865942, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:55:09,262 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34681] ipc.CallRunner(144): callId: 61 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41164 deadline: 1681527369260, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=104a1f3a98692b3503aef02f1d865942, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:09,268 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=104a1f3a98692b3503aef02f1d865942, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:55:09,268 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34681] ipc.CallRunner(144): callId: 65 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41156 deadline: 1681527369267, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=104a1f3a98692b3503aef02f1d865942, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:09,331 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=104a1f3a98692b3503aef02f1d865942, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:55:09,331 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34681] ipc.CallRunner(144): callId: 22 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41166 deadline: 1681527369327, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=104a1f3a98692b3503aef02f1d865942, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:09,332 DEBUG [Thread-2179] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4149 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=104a1f3a98692b3503aef02f1d865942, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_2' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1681527301940.104a1f3a98692b3503aef02f1d865942., hostname=jenkins-hbase20.apache.org,34681,1681527209133, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=104a1f3a98692b3503aef02f1d865942, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor31.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=104a1f3a98692b3503aef02f1d865942, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-15 02:55:09,351 INFO [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=130 (bloomFilter=true), to=hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/104a1f3a98692b3503aef02f1d865942/.tmp/B/8f11a364308f4fe0a469630c6f70f2a3 2023-04-15 02:55:09,761 INFO [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=130 (bloomFilter=true), to=hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/104a1f3a98692b3503aef02f1d865942/.tmp/C/5cb02c2a67b645b8b76212926ac890f2 2023-04-15 02:55:09,765 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/104a1f3a98692b3503aef02f1d865942/.tmp/A/93140167335e47b4970764378c84ea19 as hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/104a1f3a98692b3503aef02f1d865942/A/93140167335e47b4970764378c84ea19 2023-04-15 02:55:09,768 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=104a1f3a98692b3503aef02f1d865942, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:55:09,768 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34681] ipc.CallRunner(144): callId: 63 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41164 deadline: 1681527369763, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=104a1f3a98692b3503aef02f1d865942, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:09,770 INFO [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/104a1f3a98692b3503aef02f1d865942/A/93140167335e47b4970764378c84ea19, entries=150, sequenceid=130, filesize=25.2 K 2023-04-15 02:55:09,770 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/104a1f3a98692b3503aef02f1d865942/.tmp/B/8f11a364308f4fe0a469630c6f70f2a3 as hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/104a1f3a98692b3503aef02f1d865942/B/8f11a364308f4fe0a469630c6f70f2a3 2023-04-15 02:55:09,773 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=104a1f3a98692b3503aef02f1d865942, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:55:09,773 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34681] ipc.CallRunner(144): callId: 61 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41118 deadline: 1681527369766, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=104a1f3a98692b3503aef02f1d865942, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:09,774 INFO [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/104a1f3a98692b3503aef02f1d865942/B/8f11a364308f4fe0a469630c6f70f2a3, entries=150, sequenceid=130, filesize=11.7 K 2023-04-15 02:55:09,777 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/104a1f3a98692b3503aef02f1d865942/.tmp/C/5cb02c2a67b645b8b76212926ac890f2 as hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/104a1f3a98692b3503aef02f1d865942/C/5cb02c2a67b645b8b76212926ac890f2 2023-04-15 02:55:09,778 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=104a1f3a98692b3503aef02f1d865942, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:55:09,779 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34681] ipc.CallRunner(144): callId: 67 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41156 deadline: 1681527369771, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=104a1f3a98692b3503aef02f1d865942, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:09,781 INFO [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/104a1f3a98692b3503aef02f1d865942/C/5cb02c2a67b645b8b76212926ac890f2, entries=150, sequenceid=130, filesize=11.7 K 2023-04-15 02:55:09,784 INFO [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~134.18 KB/137400, heapSize ~366.19 KB/374976, currentSize=60.38 KB/61830 for 104a1f3a98692b3503aef02f1d865942 in 1254ms, sequenceid=130, compaction requested=true 2023-04-15 02:55:09,784 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 104a1f3a98692b3503aef02f1d865942: 2023-04-15 02:55:09,784 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681527301940.104a1f3a98692b3503aef02f1d865942. 2023-04-15 02:55:09,784 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-15 02:55:09,785 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-15 02:55:09,785 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:09,785 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-15 02:55:09,785 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34681,1681527209133' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-15 02:55:09,786 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:09,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:09,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:55:09,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:55:09,786 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:09,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:55:09,786 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:09,786 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-15 02:55:09,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:55:09,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:55:09,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:09,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:55:09,787 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34681,1681527209133' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-15 02:55:09,787 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@724619aa[Count = 0] remaining members to acquire global barrier 2023-04-15 02:55:09,787 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-15 02:55:09,787 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:09,788 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:09,788 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:09,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:09,788 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-15 02:55:09,788 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-15 02:55:09,788 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34681,1681527209133' in zk 2023-04-15 02:55:09,788 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:09,788 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-15 02:55:09,789 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:09,789 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-15 02:55:09,789 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-15 02:55:09,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:09,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:55:09,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:55:09,789 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-15 02:55:09,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:55:09,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:55:09,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:55:09,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:09,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:55:09,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:55:09,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:09,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34681,1681527209133': 2023-04-15 02:55:09,791 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34681,1681527209133' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-15 02:55:09,791 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-15 02:55:09,791 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-15 02:55:09,791 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-15 02:55:09,791 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-15 02:55:09,792 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-15 02:55:09,792 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:09,792 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:09,793 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-15 02:55:09,793 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:09,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:09,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:55:09,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:55:09,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:09,793 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-15 02:55:09,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:55:09,793 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:09,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-15 02:55:09,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:55:09,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:09,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:55:09,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:55:09,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:09,795 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:09,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:55:09,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:55:09,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:09,797 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-15 02:55:09,797 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-15 02:55:09,797 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-15 02:55:09,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-15 02:55:09,797 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-15 02:55:09,797 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:09,797 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:55:09,797 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:55:09,797 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:09,797 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:09,797 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:09,797 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:09,797 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-15 02:55:09,797 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-15 02:55:09,797 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-15 02:55:09,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-15 02:55:09,798 DEBUG [Thread-2181] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-15 02:55:09,798 DEBUG [Thread-2181] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-15 02:55:09,898 DEBUG [Thread-2181] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-15 02:55:09,898 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-15 02:55:09,899 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-15 02:55:09,900 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-15 02:55:09,900 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-15 02:55:09,900 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-15 02:55:09,900 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-15 02:55:09,900 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-15 02:55:09,901 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-15 02:55:09,901 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:09,901 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:55:09,902 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:09,902 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-15 02:55:09,902 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:09,902 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-15 02:55:09,903 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-15 02:55:09,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-15 02:55:09,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:55:09,904 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:09,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-15 02:55:09,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:55:09,904 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-15 02:55:09,904 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-15 02:55:09,905 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-15 02:55:09,905 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-15 02:55:09,905 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-15 02:55:09,905 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-15 02:55:09,905 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-15 02:55:09,905 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-15 02:55:09,905 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681527301940.104a1f3a98692b3503aef02f1d865942. 2023-04-15 02:55:09,905 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681527301940.104a1f3a98692b3503aef02f1d865942. started... 2023-04-15 02:55:09,905 INFO [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 104a1f3a98692b3503aef02f1d865942 3/3 column families, dataSize=60.38 KB heapSize=180.05 KB 2023-04-15 02:55:09,905 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 104a1f3a98692b3503aef02f1d865942, store=A 2023-04-15 02:55:09,905 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-15 02:55:09,905 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 104a1f3a98692b3503aef02f1d865942, store=B 2023-04-15 02:55:09,905 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-15 02:55:09,906 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 104a1f3a98692b3503aef02f1d865942, store=C 2023-04-15 02:55:09,906 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-15 02:55:10,154 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34681] regionserver.HRegion(9158): Flush requested on 104a1f3a98692b3503aef02f1d865942 2023-04-15 02:55:10,154 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681527301940.104a1f3a98692b3503aef02f1d865942. as already flushing 2023-04-15 02:55:10,273 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=104a1f3a98692b3503aef02f1d865942, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:55:10,273 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34681] ipc.CallRunner(144): callId: 80 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41180 deadline: 1681527370264, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=104a1f3a98692b3503aef02f1d865942, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:10,326 INFO [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230415306cff40458d437c819e48ab33271a62 to hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230415306cff40458d437c819e48ab33271a62 2023-04-15 02:55:10,344 INFO [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=142, memsize=20.1 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/104a1f3a98692b3503aef02f1d865942/.tmp/A/3127ee9695b9478896a251390e068af3 2023-04-15 02:55:10,381 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=104a1f3a98692b3503aef02f1d865942, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:55:10,381 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34681] ipc.CallRunner(144): callId: 82 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41180 deadline: 1681527370374, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=104a1f3a98692b3503aef02f1d865942, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:10,587 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=104a1f3a98692b3503aef02f1d865942, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:55:10,587 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34681] ipc.CallRunner(144): callId: 84 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41180 deadline: 1681527370582, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=104a1f3a98692b3503aef02f1d865942, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:10,768 INFO [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=142 (bloomFilter=true), to=hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/104a1f3a98692b3503aef02f1d865942/.tmp/B/ed4f08d05dca469d86fa7767faadf370 2023-04-15 02:55:10,782 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=104a1f3a98692b3503aef02f1d865942, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:55:10,782 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34681] ipc.CallRunner(144): callId: 65 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41164 deadline: 1681527370776, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=104a1f3a98692b3503aef02f1d865942, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:10,782 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=104a1f3a98692b3503aef02f1d865942, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:55:10,782 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34681] ipc.CallRunner(144): callId: 69 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41156 deadline: 1681527370779, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=104a1f3a98692b3503aef02f1d865942, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:10,782 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=104a1f3a98692b3503aef02f1d865942, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:55:10,782 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34681] ipc.CallRunner(144): callId: 63 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41118 deadline: 1681527370781, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=104a1f3a98692b3503aef02f1d865942, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:10,893 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=104a1f3a98692b3503aef02f1d865942, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:55:10,893 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34681] ipc.CallRunner(144): callId: 86 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41180 deadline: 1681527370889, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=104a1f3a98692b3503aef02f1d865942, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:11,179 INFO [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=142 (bloomFilter=true), to=hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/104a1f3a98692b3503aef02f1d865942/.tmp/C/fa4a013e158245edab7ec80baac43f81 2023-04-15 02:55:11,182 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/104a1f3a98692b3503aef02f1d865942/.tmp/A/3127ee9695b9478896a251390e068af3 as hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/104a1f3a98692b3503aef02f1d865942/A/3127ee9695b9478896a251390e068af3 2023-04-15 02:55:11,185 INFO [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/104a1f3a98692b3503aef02f1d865942/A/3127ee9695b9478896a251390e068af3, entries=150, sequenceid=142, filesize=25.3 K 2023-04-15 02:55:11,186 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/104a1f3a98692b3503aef02f1d865942/.tmp/B/ed4f08d05dca469d86fa7767faadf370 as hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/104a1f3a98692b3503aef02f1d865942/B/ed4f08d05dca469d86fa7767faadf370 2023-04-15 02:55:11,189 INFO [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/104a1f3a98692b3503aef02f1d865942/B/ed4f08d05dca469d86fa7767faadf370, entries=150, sequenceid=142, filesize=11.8 K 2023-04-15 02:55:11,190 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.StoreScanner(961): StoreScanner already closing. There is no need to updateReaders 2023-04-15 02:55:11,190 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/104a1f3a98692b3503aef02f1d865942/.tmp/C/fa4a013e158245edab7ec80baac43f81 as hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/104a1f3a98692b3503aef02f1d865942/C/fa4a013e158245edab7ec80baac43f81 2023-04-15 02:55:11,196 INFO [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/104a1f3a98692b3503aef02f1d865942/C/fa4a013e158245edab7ec80baac43f81, entries=150, sequenceid=142, filesize=11.8 K 2023-04-15 02:55:11,197 INFO [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~60.38 KB/61830, heapSize ~180 KB/184320, currentSize=114.05 KB/116790 for 104a1f3a98692b3503aef02f1d865942 in 1292ms, sequenceid=142, compaction requested=true 2023-04-15 02:55:11,197 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 104a1f3a98692b3503aef02f1d865942: 2023-04-15 02:55:11,197 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681527301940.104a1f3a98692b3503aef02f1d865942. 2023-04-15 02:55:11,197 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-15 02:55:11,197 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-15 02:55:11,197 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:11,197 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-15 02:55:11,197 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34681,1681527209133' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-15 02:55:11,198 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:11,198 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:11,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:11,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:55:11,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:55:11,198 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:11,198 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-15 02:55:11,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:55:11,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:55:11,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:55:11,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:11,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:55:11,199 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34681,1681527209133' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-15 02:55:11,199 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@47fe096e[Count = 0] remaining members to acquire global barrier 2023-04-15 02:55:11,199 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-15 02:55:11,199 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:11,200 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:11,200 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:11,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:11,200 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-15 02:55:11,200 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-15 02:55:11,200 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34681,1681527209133' in zk 2023-04-15 02:55:11,200 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:11,200 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-15 02:55:11,200 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:11,200 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-15 02:55:11,201 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-15 02:55:11,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:11,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:55:11,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:55:11,201 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-15 02:55:11,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:55:11,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:55:11,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:55:11,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:11,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:55:11,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:55:11,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:11,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34681,1681527209133': 2023-04-15 02:55:11,202 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34681,1681527209133' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-15 02:55:11,202 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-15 02:55:11,202 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-15 02:55:11,202 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-15 02:55:11,202 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-15 02:55:11,202 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-15 02:55:11,203 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:11,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:11,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:55:11,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:55:11,203 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:11,203 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-15 02:55:11,203 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:11,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:11,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:55:11,203 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-15 02:55:11,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-15 02:55:11,203 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:11,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:55:11,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:11,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:55:11,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:55:11,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:11,204 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:11,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:55:11,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:55:11,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:11,205 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:11,205 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:55:11,205 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-15 02:55:11,206 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:55:11,206 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-15 02:55:11,206 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-15 02:55:11,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-15 02:55:11,206 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-15 02:55:11,206 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:11,206 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-15 02:55:11,206 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-15 02:55:11,206 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-15 02:55:11,206 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-15 02:55:11,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-15 02:55:11,206 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:11,206 DEBUG [Thread-2181] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-15 02:55:11,206 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:11,206 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:11,206 DEBUG [Thread-2181] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-15 02:55:11,306 DEBUG [Thread-2181] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-15 02:55:11,307 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-15 02:55:11,308 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-15 02:55:11,309 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-15 02:55:11,309 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-15 02:55:11,309 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-15 02:55:11,309 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-15 02:55:11,309 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-15 02:55:11,309 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-15 02:55:11,309 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:11,309 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:55:11,310 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:11,310 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-15 02:55:11,310 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-15 02:55:11,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-15 02:55:11,310 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:11,310 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-15 02:55:11,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:55:11,311 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:11,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-15 02:55:11,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:55:11,311 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-15 02:55:11,311 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-15 02:55:11,311 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-15 02:55:11,311 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-15 02:55:11,311 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-15 02:55:11,311 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-15 02:55:11,312 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-15 02:55:11,312 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-15 02:55:11,312 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681527301940.104a1f3a98692b3503aef02f1d865942. 2023-04-15 02:55:11,312 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681527301940.104a1f3a98692b3503aef02f1d865942. started... 2023-04-15 02:55:11,312 INFO [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 104a1f3a98692b3503aef02f1d865942 3/3 column families, dataSize=114.05 KB heapSize=339.42 KB 2023-04-15 02:55:11,312 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 104a1f3a98692b3503aef02f1d865942, store=A 2023-04-15 02:55:11,312 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-15 02:55:11,312 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 104a1f3a98692b3503aef02f1d865942, store=B 2023-04-15 02:55:11,312 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-15 02:55:11,312 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 104a1f3a98692b3503aef02f1d865942, store=C 2023-04-15 02:55:11,312 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-15 02:55:11,397 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34681] regionserver.HRegion(9158): Flush requested on 104a1f3a98692b3503aef02f1d865942 2023-04-15 02:55:11,397 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681527301940.104a1f3a98692b3503aef02f1d865942. as already flushing 2023-04-15 02:55:11,450 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=104a1f3a98692b3503aef02f1d865942, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:55:11,450 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34681] ipc.CallRunner(144): callId: 97 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41180 deadline: 1681527371444, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=104a1f3a98692b3503aef02f1d865942, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:11,556 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=104a1f3a98692b3503aef02f1d865942, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:55:11,556 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34681] ipc.CallRunner(144): callId: 99 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41180 deadline: 1681527371551, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=104a1f3a98692b3503aef02f1d865942, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:11,734 INFO [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023041516dbfe40ca474e9098188aadd6ddf7ca to hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023041516dbfe40ca474e9098188aadd6ddf7ca 2023-04-15 02:55:11,760 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=104a1f3a98692b3503aef02f1d865942, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:55:11,760 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34681] ipc.CallRunner(144): callId: 101 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41180 deadline: 1681527371758, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=104a1f3a98692b3503aef02f1d865942, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:12,066 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=104a1f3a98692b3503aef02f1d865942, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:55:12,066 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34681] ipc.CallRunner(144): callId: 103 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41180 deadline: 1681527372061, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=104a1f3a98692b3503aef02f1d865942, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:12,138 INFO [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=162, memsize=38.0 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/104a1f3a98692b3503aef02f1d865942/.tmp/A/6efb8c0a53064a5cbeec795d2cb8faf7 2023-04-15 02:55:12,147 INFO [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=162 (bloomFilter=true), to=hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/104a1f3a98692b3503aef02f1d865942/.tmp/B/aad59967db1a4dcd8841e8dca8885b2a 2023-04-15 02:55:12,155 INFO [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=162 (bloomFilter=true), to=hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/104a1f3a98692b3503aef02f1d865942/.tmp/C/62fe15263bef44aea7b83db93c46c4f9 2023-04-15 02:55:12,158 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/104a1f3a98692b3503aef02f1d865942/.tmp/A/6efb8c0a53064a5cbeec795d2cb8faf7 as hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/104a1f3a98692b3503aef02f1d865942/A/6efb8c0a53064a5cbeec795d2cb8faf7 2023-04-15 02:55:12,161 INFO [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/104a1f3a98692b3503aef02f1d865942/A/6efb8c0a53064a5cbeec795d2cb8faf7, entries=150, sequenceid=162, filesize=25.3 K 2023-04-15 02:55:12,161 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/104a1f3a98692b3503aef02f1d865942/.tmp/B/aad59967db1a4dcd8841e8dca8885b2a as hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/104a1f3a98692b3503aef02f1d865942/B/aad59967db1a4dcd8841e8dca8885b2a 2023-04-15 02:55:12,165 INFO [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/104a1f3a98692b3503aef02f1d865942/B/aad59967db1a4dcd8841e8dca8885b2a, entries=150, sequenceid=162, filesize=11.8 K 2023-04-15 02:55:12,166 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/104a1f3a98692b3503aef02f1d865942/.tmp/C/62fe15263bef44aea7b83db93c46c4f9 as hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/104a1f3a98692b3503aef02f1d865942/C/62fe15263bef44aea7b83db93c46c4f9 2023-04-15 02:55:12,170 INFO [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/104a1f3a98692b3503aef02f1d865942/C/62fe15263bef44aea7b83db93c46c4f9, entries=150, sequenceid=162, filesize=11.8 K 2023-04-15 02:55:12,171 INFO [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~114.05 KB/116790, heapSize ~339.38 KB/347520, currentSize=60.38 KB/61830 for 104a1f3a98692b3503aef02f1d865942 in 859ms, sequenceid=162, compaction requested=true 2023-04-15 02:55:12,171 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 104a1f3a98692b3503aef02f1d865942: 2023-04-15 02:55:12,171 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681527301940.104a1f3a98692b3503aef02f1d865942. 2023-04-15 02:55:12,171 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-15 02:55:12,171 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-15 02:55:12,171 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:12,171 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-15 02:55:12,171 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34681,1681527209133' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-15 02:55:12,172 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:12,172 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:12,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:12,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:55:12,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:55:12,172 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:12,172 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-15 02:55:12,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:55:12,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:55:12,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:55:12,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:12,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:55:12,175 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34681,1681527209133' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-15 02:55:12,175 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@368ff763[Count = 0] remaining members to acquire global barrier 2023-04-15 02:55:12,175 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-15 02:55:12,175 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:12,175 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:12,175 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:12,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:12,175 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-15 02:55:12,175 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-15 02:55:12,175 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34681,1681527209133' in zk 2023-04-15 02:55:12,175 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:12,175 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-15 02:55:12,176 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-15 02:55:12,176 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-15 02:55:12,176 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:12,176 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-15 02:55:12,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:12,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:55:12,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:55:12,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:55:12,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:55:12,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:55:12,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:12,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:55:12,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:55:12,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:12,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34681,1681527209133': 2023-04-15 02:55:12,178 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34681,1681527209133' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-15 02:55:12,178 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-15 02:55:12,178 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-15 02:55:12,178 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-15 02:55:12,178 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-15 02:55:12,178 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-15 02:55:12,179 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:12,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:12,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:55:12,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:55:12,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:55:12,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:55:12,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:55:12,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:55:12,180 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:12,180 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-15 02:55:12,180 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:12,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:12,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:12,181 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-15 02:55:12,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-15 02:55:12,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:55:12,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:12,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:55:12,181 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:12,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:12,182 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:12,190 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-15 02:55:12,190 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:12,190 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-15 02:55:12,190 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-15 02:55:12,190 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:55:12,190 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-15 02:55:12,190 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-15 02:55:12,190 DEBUG [Thread-2181] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-15 02:55:12,190 DEBUG [Thread-2181] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-15 02:55:12,190 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-15 02:55:12,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-15 02:55:12,190 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:55:12,191 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:12,191 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:12,191 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:12,191 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:12,192 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-15 02:55:12,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-15 02:55:12,291 DEBUG [Thread-2181] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-15 02:55:12,291 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-15 02:55:12,292 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-15 02:55:12,293 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-15 02:55:12,294 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-15 02:55:12,294 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-15 02:55:12,294 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-15 02:55:12,294 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-15 02:55:12,294 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-15 02:55:12,294 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:12,294 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:55:12,295 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:12,296 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:12,296 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-15 02:55:12,298 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-15 02:55:12,298 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-15 02:55:12,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-15 02:55:12,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:55:12,299 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:12,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-15 02:55:12,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:55:12,299 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-15 02:55:12,299 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-15 02:55:12,299 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-15 02:55:12,299 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-15 02:55:12,299 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-15 02:55:12,300 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-15 02:55:12,302 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-15 02:55:12,302 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-15 02:55:12,302 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681527301940.104a1f3a98692b3503aef02f1d865942. 2023-04-15 02:55:12,302 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681527301940.104a1f3a98692b3503aef02f1d865942. started... 2023-04-15 02:55:12,302 INFO [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 104a1f3a98692b3503aef02f1d865942 3/3 column families, dataSize=60.38 KB heapSize=180.05 KB 2023-04-15 02:55:12,302 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 104a1f3a98692b3503aef02f1d865942, store=A 2023-04-15 02:55:12,302 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-15 02:55:12,302 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 104a1f3a98692b3503aef02f1d865942, store=B 2023-04-15 02:55:12,302 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-15 02:55:12,302 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 104a1f3a98692b3503aef02f1d865942, store=C 2023-04-15 02:55:12,303 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-15 02:55:12,571 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34681] regionserver.HRegion(9158): Flush requested on 104a1f3a98692b3503aef02f1d865942 2023-04-15 02:55:12,571 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681527301940.104a1f3a98692b3503aef02f1d865942. as already flushing 2023-04-15 02:55:12,700 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=104a1f3a98692b3503aef02f1d865942, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:55:12,700 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34681] ipc.CallRunner(144): callId: 122 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41180 deadline: 1681527372693, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=104a1f3a98692b3503aef02f1d865942, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:12,723 INFO [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230415562c08ea00f74d6493b762871d68dc04 to hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230415562c08ea00f74d6493b762871d68dc04 2023-04-15 02:55:12,798 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=104a1f3a98692b3503aef02f1d865942, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:55:12,798 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34681] ipc.CallRunner(144): callId: 65 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41118 deadline: 1681527372792, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=104a1f3a98692b3503aef02f1d865942, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:12,799 DEBUG [Thread-2173] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4162 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=104a1f3a98692b3503aef02f1d865942, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_0' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1681527301940.104a1f3a98692b3503aef02f1d865942., hostname=jenkins-hbase20.apache.org,34681,1681527209133, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=104a1f3a98692b3503aef02f1d865942, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor31.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=104a1f3a98692b3503aef02f1d865942, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-15 02:55:12,802 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=104a1f3a98692b3503aef02f1d865942, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:55:12,802 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34681] ipc.CallRunner(144): callId: 67 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41164 deadline: 1681527372796, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=104a1f3a98692b3503aef02f1d865942, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:12,803 DEBUG [Thread-2177] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4163 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=104a1f3a98692b3503aef02f1d865942, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_1' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1681527301940.104a1f3a98692b3503aef02f1d865942., hostname=jenkins-hbase20.apache.org,34681,1681527209133, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=104a1f3a98692b3503aef02f1d865942, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor31.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=104a1f3a98692b3503aef02f1d865942, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-15 02:55:12,806 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=104a1f3a98692b3503aef02f1d865942, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:55:12,806 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34681] ipc.CallRunner(144): callId: 71 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41156 deadline: 1681527372800, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=104a1f3a98692b3503aef02f1d865942, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:12,807 DEBUG [Thread-2171] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4165 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=104a1f3a98692b3503aef02f1d865942, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_1' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1681527301940.104a1f3a98692b3503aef02f1d865942., hostname=jenkins-hbase20.apache.org,34681,1681527209133, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=104a1f3a98692b3503aef02f1d865942, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor31.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=104a1f3a98692b3503aef02f1d865942, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-15 02:55:12,808 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=104a1f3a98692b3503aef02f1d865942, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:55:12,808 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34681] ipc.CallRunner(144): callId: 124 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41180 deadline: 1681527372801, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=104a1f3a98692b3503aef02f1d865942, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:13,013 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=104a1f3a98692b3503aef02f1d865942, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:55:13,013 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34681] ipc.CallRunner(144): callId: 126 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41180 deadline: 1681527373009, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=104a1f3a98692b3503aef02f1d865942, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:13,127 INFO [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=174, memsize=20.1 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/104a1f3a98692b3503aef02f1d865942/.tmp/A/ee83bff49f4a41f08854c29cf00b177d 2023-04-15 02:55:13,138 INFO [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=174 (bloomFilter=true), to=hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/104a1f3a98692b3503aef02f1d865942/.tmp/B/0b220438b5504a3b93e3041e42f9f358 2023-04-15 02:55:13,317 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=104a1f3a98692b3503aef02f1d865942, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:55:13,318 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34681] ipc.CallRunner(144): callId: 128 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:41180 deadline: 1681527373315, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=104a1f3a98692b3503aef02f1d865942, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:13,353 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=104a1f3a98692b3503aef02f1d865942, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:55:13,354 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34681] ipc.CallRunner(144): callId: 24 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41166 deadline: 1681527373348, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=104a1f3a98692b3503aef02f1d865942, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:13,355 DEBUG [Thread-2179] client.RpcRetryingCallerImpl(129): Call exception, tries=7, retries=16, started=8172 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=104a1f3a98692b3503aef02f1d865942, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_2' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1681527301940.104a1f3a98692b3503aef02f1d865942., hostname=jenkins-hbase20.apache.org,34681,1681527209133, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=104a1f3a98692b3503aef02f1d865942, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor31.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=104a1f3a98692b3503aef02f1d865942, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-15 02:55:13,546 INFO [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=174 (bloomFilter=true), to=hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/104a1f3a98692b3503aef02f1d865942/.tmp/C/8cf30cb2449c4d4c915a6fed58dbb6cb 2023-04-15 02:55:13,550 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/104a1f3a98692b3503aef02f1d865942/.tmp/A/ee83bff49f4a41f08854c29cf00b177d as hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/104a1f3a98692b3503aef02f1d865942/A/ee83bff49f4a41f08854c29cf00b177d 2023-04-15 02:55:13,553 INFO [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/104a1f3a98692b3503aef02f1d865942/A/ee83bff49f4a41f08854c29cf00b177d, entries=150, sequenceid=174, filesize=25.3 K 2023-04-15 02:55:13,554 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/104a1f3a98692b3503aef02f1d865942/.tmp/B/0b220438b5504a3b93e3041e42f9f358 as hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/104a1f3a98692b3503aef02f1d865942/B/0b220438b5504a3b93e3041e42f9f358 2023-04-15 02:55:13,558 INFO [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/104a1f3a98692b3503aef02f1d865942/B/0b220438b5504a3b93e3041e42f9f358, entries=150, sequenceid=174, filesize=11.8 K 2023-04-15 02:55:13,559 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/104a1f3a98692b3503aef02f1d865942/.tmp/C/8cf30cb2449c4d4c915a6fed58dbb6cb as hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/104a1f3a98692b3503aef02f1d865942/C/8cf30cb2449c4d4c915a6fed58dbb6cb 2023-04-15 02:55:13,568 INFO [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/104a1f3a98692b3503aef02f1d865942/C/8cf30cb2449c4d4c915a6fed58dbb6cb, entries=150, sequenceid=174, filesize=11.8 K 2023-04-15 02:55:13,568 INFO [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~60.38 KB/61830, heapSize ~180 KB/184320, currentSize=114.05 KB/116790 for 104a1f3a98692b3503aef02f1d865942 in 1266ms, sequenceid=174, compaction requested=true 2023-04-15 02:55:13,568 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 104a1f3a98692b3503aef02f1d865942: 2023-04-15 02:55:13,568 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681527301940.104a1f3a98692b3503aef02f1d865942. 2023-04-15 02:55:13,568 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-15 02:55:13,568 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-15 02:55:13,569 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:13,569 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-15 02:55:13,569 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34681,1681527209133' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-15 02:55:13,569 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:13,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:13,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:55:13,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:55:13,570 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:13,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:55:13,570 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:13,570 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-15 02:55:13,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:55:13,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:55:13,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:13,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:55:13,571 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34681,1681527209133' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-15 02:55:13,571 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4c9913f1[Count = 0] remaining members to acquire global barrier 2023-04-15 02:55:13,571 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-15 02:55:13,571 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:13,572 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:13,572 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:13,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:13,572 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:13,572 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-15 02:55:13,572 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-15 02:55:13,572 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-15 02:55:13,572 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34681,1681527209133' in zk 2023-04-15 02:55:13,572 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:13,572 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-15 02:55:13,573 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-15 02:55:13,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:13,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:55:13,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:55:13,573 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-15 02:55:13,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:55:13,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:55:13,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:55:13,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:13,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:55:13,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:55:13,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:13,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34681,1681527209133': 2023-04-15 02:55:13,574 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34681,1681527209133' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-15 02:55:13,574 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-15 02:55:13,574 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-15 02:55:13,574 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-15 02:55:13,574 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-15 02:55:13,574 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-15 02:55:13,575 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:13,575 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:13,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:13,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:55:13,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:55:13,575 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:13,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:13,575 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-15 02:55:13,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:55:13,576 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-15 02:55:13,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-15 02:55:13,576 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:13,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:13,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:55:13,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:55:13,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:55:13,577 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:13,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:13,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:55:13,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:55:13,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:13,578 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-15 02:55:13,578 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:13,578 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-15 02:55:13,578 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-15 02:55:13,578 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-15 02:55:13,578 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-15 02:55:13,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-15 02:55:13,578 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:55:13,578 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:55:13,579 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:13,579 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:13,579 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:13,579 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:13,578 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-15 02:55:13,579 DEBUG [Thread-2181] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-15 02:55:13,578 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-15 02:55:13,579 DEBUG [Thread-2181] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-15 02:55:13,579 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-15 02:55:13,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-15 02:55:13,679 DEBUG [Thread-2181] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-15 02:55:13,679 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-15 02:55:13,681 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-15 02:55:13,682 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-15 02:55:13,682 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-15 02:55:13,682 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-15 02:55:13,682 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-15 02:55:13,682 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-15 02:55:13,682 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-15 02:55:13,682 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:13,682 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:55:13,683 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:13,683 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-15 02:55:13,683 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-15 02:55:13,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-15 02:55:13,683 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:13,683 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-15 02:55:13,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:55:13,683 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:13,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-15 02:55:13,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:55:13,684 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-15 02:55:13,684 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-15 02:55:13,684 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-15 02:55:13,684 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-15 02:55:13,684 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-15 02:55:13,684 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-15 02:55:13,684 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-15 02:55:13,684 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-15 02:55:13,684 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681527301940.104a1f3a98692b3503aef02f1d865942. 2023-04-15 02:55:13,684 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681527301940.104a1f3a98692b3503aef02f1d865942. started... 2023-04-15 02:55:13,684 INFO [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 104a1f3a98692b3503aef02f1d865942 3/3 column families, dataSize=114.05 KB heapSize=339.42 KB 2023-04-15 02:55:13,685 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 104a1f3a98692b3503aef02f1d865942, store=A 2023-04-15 02:55:13,685 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-15 02:55:13,685 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 104a1f3a98692b3503aef02f1d865942, store=B 2023-04-15 02:55:13,685 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-15 02:55:13,685 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 104a1f3a98692b3503aef02f1d865942, store=C 2023-04-15 02:55:13,685 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-15 02:55:13,827 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34681] regionserver.HRegion(9158): Flush requested on 104a1f3a98692b3503aef02f1d865942 2023-04-15 02:55:13,827 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681527301940.104a1f3a98692b3503aef02f1d865942. as already flushing 2023-04-15 02:55:13,884 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=104a1f3a98692b3503aef02f1d865942, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:55:13,884 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34681] ipc.CallRunner(144): callId: 139 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:41180 deadline: 1681527373883, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=104a1f3a98692b3503aef02f1d865942, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:13,989 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=104a1f3a98692b3503aef02f1d865942, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:55:13,989 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34681] ipc.CallRunner(144): callId: 141 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:41180 deadline: 1681527373985, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=104a1f3a98692b3503aef02f1d865942, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:14,096 INFO [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304152be756929c244a4b861d3dbeee4d51d3 to hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304152be756929c244a4b861d3dbeee4d51d3 2023-04-15 02:55:14,197 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=104a1f3a98692b3503aef02f1d865942, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:55:14,197 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34681] ipc.CallRunner(144): callId: 143 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:41180 deadline: 1681527374191, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=104a1f3a98692b3503aef02f1d865942, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:14,501 INFO [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=194, memsize=38.0 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/104a1f3a98692b3503aef02f1d865942/.tmp/A/10794015f4a84f0eb0bd35ac69adbe9e 2023-04-15 02:55:14,503 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=104a1f3a98692b3503aef02f1d865942, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:55:14,503 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34681] ipc.CallRunner(144): callId: 145 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:41180 deadline: 1681527374499, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=104a1f3a98692b3503aef02f1d865942, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:14,911 INFO [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=194 (bloomFilter=true), to=hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/104a1f3a98692b3503aef02f1d865942/.tmp/B/f5361327d59449549ed8db22f4e18551 2023-04-15 02:55:15,012 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=104a1f3a98692b3503aef02f1d865942, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:55:15,012 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34681] ipc.CallRunner(144): callId: 147 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:41180 deadline: 1681527375006, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=104a1f3a98692b3503aef02f1d865942, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:15,326 INFO [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=194 (bloomFilter=true), to=hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/104a1f3a98692b3503aef02f1d865942/.tmp/C/4502e62cc1574f8bb74a3859a770227c 2023-04-15 02:55:15,330 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/104a1f3a98692b3503aef02f1d865942/.tmp/A/10794015f4a84f0eb0bd35ac69adbe9e as hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/104a1f3a98692b3503aef02f1d865942/A/10794015f4a84f0eb0bd35ac69adbe9e 2023-04-15 02:55:15,332 INFO [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/104a1f3a98692b3503aef02f1d865942/A/10794015f4a84f0eb0bd35ac69adbe9e, entries=150, sequenceid=194, filesize=25.3 K 2023-04-15 02:55:15,333 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/104a1f3a98692b3503aef02f1d865942/.tmp/B/f5361327d59449549ed8db22f4e18551 as hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/104a1f3a98692b3503aef02f1d865942/B/f5361327d59449549ed8db22f4e18551 2023-04-15 02:55:15,336 INFO [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/104a1f3a98692b3503aef02f1d865942/B/f5361327d59449549ed8db22f4e18551, entries=150, sequenceid=194, filesize=11.8 K 2023-04-15 02:55:15,337 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/104a1f3a98692b3503aef02f1d865942/.tmp/C/4502e62cc1574f8bb74a3859a770227c as hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/104a1f3a98692b3503aef02f1d865942/C/4502e62cc1574f8bb74a3859a770227c 2023-04-15 02:55:15,339 INFO [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/104a1f3a98692b3503aef02f1d865942/C/4502e62cc1574f8bb74a3859a770227c, entries=150, sequenceid=194, filesize=11.8 K 2023-04-15 02:55:15,340 INFO [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~114.05 KB/116790, heapSize ~339.38 KB/347520, currentSize=60.38 KB/61830 for 104a1f3a98692b3503aef02f1d865942 in 1656ms, sequenceid=194, compaction requested=true 2023-04-15 02:55:15,340 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 104a1f3a98692b3503aef02f1d865942: 2023-04-15 02:55:15,340 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681527301940.104a1f3a98692b3503aef02f1d865942. 2023-04-15 02:55:15,340 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-15 02:55:15,340 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-15 02:55:15,340 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:15,340 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-15 02:55:15,340 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34681,1681527209133' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-15 02:55:15,341 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:15,341 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:15,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:15,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:55:15,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:55:15,341 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:15,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:55:15,341 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-15 02:55:15,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:55:15,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:55:15,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:15,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:55:15,343 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34681,1681527209133' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-15 02:55:15,343 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@ed0d881[Count = 0] remaining members to acquire global barrier 2023-04-15 02:55:15,343 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-15 02:55:15,343 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:15,343 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:15,343 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:15,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:15,343 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-15 02:55:15,343 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-15 02:55:15,343 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34681,1681527209133' in zk 2023-04-15 02:55:15,343 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:15,343 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-15 02:55:15,344 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-15 02:55:15,344 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:15,344 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-15 02:55:15,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:15,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:55:15,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:55:15,344 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-15 02:55:15,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:55:15,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:55:15,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:55:15,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:15,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:55:15,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:55:15,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:15,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34681,1681527209133': 2023-04-15 02:55:15,346 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34681,1681527209133' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-15 02:55:15,346 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-15 02:55:15,346 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-15 02:55:15,346 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-15 02:55:15,346 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-15 02:55:15,346 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-15 02:55:15,347 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:15,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:15,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:55:15,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:55:15,347 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:15,347 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-15 02:55:15,347 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:15,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:15,347 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-15 02:55:15,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-15 02:55:15,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:55:15,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:15,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:55:15,348 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:15,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:55:15,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:55:15,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:15,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:55:15,349 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:15,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:55:15,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:15,350 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:15,350 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-15 02:55:15,350 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:55:15,350 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-15 02:55:15,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-15 02:55:15,350 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-15 02:55:15,350 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-15 02:55:15,350 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-15 02:55:15,350 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-15 02:55:15,350 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:55:15,350 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:15,350 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:15,350 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-15 02:55:15,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-15 02:55:15,350 DEBUG [Thread-2181] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-15 02:55:15,350 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:15,351 DEBUG [Thread-2181] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-15 02:55:15,351 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:15,451 DEBUG [Thread-2181] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-15 02:55:15,451 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-15 02:55:15,452 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-15 02:55:15,454 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-15 02:55:15,454 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-15 02:55:15,454 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-15 02:55:15,454 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-15 02:55:15,454 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-15 02:55:15,454 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-15 02:55:15,454 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:15,454 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:55:15,455 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-15 02:55:15,455 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:15,455 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-15 02:55:15,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-15 02:55:15,455 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:15,455 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-15 02:55:15,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:55:15,456 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:15,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-15 02:55:15,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:55:15,456 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-15 02:55:15,456 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-15 02:55:15,456 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-15 02:55:15,456 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-15 02:55:15,456 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-15 02:55:15,456 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-15 02:55:15,456 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-15 02:55:15,456 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-15 02:55:15,456 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681527301940.104a1f3a98692b3503aef02f1d865942. 2023-04-15 02:55:15,456 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681527301940.104a1f3a98692b3503aef02f1d865942. started... 2023-04-15 02:55:15,457 INFO [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 104a1f3a98692b3503aef02f1d865942 3/3 column families, dataSize=60.38 KB heapSize=180.05 KB 2023-04-15 02:55:15,457 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 104a1f3a98692b3503aef02f1d865942, store=A 2023-04-15 02:55:15,457 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-15 02:55:15,457 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 104a1f3a98692b3503aef02f1d865942, store=B 2023-04-15 02:55:15,457 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-15 02:55:15,457 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 104a1f3a98692b3503aef02f1d865942, store=C 2023-04-15 02:55:15,457 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-15 02:55:15,873 INFO [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023041510bf33b278134f818061a3b7591a9811 to hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023041510bf33b278134f818061a3b7591a9811 2023-04-15 02:55:16,027 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34681] regionserver.HRegion(9158): Flush requested on 104a1f3a98692b3503aef02f1d865942 2023-04-15 02:55:16,027 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681527301940.104a1f3a98692b3503aef02f1d865942. as already flushing 2023-04-15 02:55:16,201 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=104a1f3a98692b3503aef02f1d865942, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:55:16,201 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34681] ipc.CallRunner(144): callId: 166 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:41180 deadline: 1681527376193, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=104a1f3a98692b3503aef02f1d865942, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:16,282 INFO [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=206, memsize=20.1 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/104a1f3a98692b3503aef02f1d865942/.tmp/A/ec5f7a40bd284b86a7878a2aaef40b61 2023-04-15 02:55:16,304 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=104a1f3a98692b3503aef02f1d865942, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:55:16,304 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34681] ipc.CallRunner(144): callId: 168 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:41180 deadline: 1681527376302, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=104a1f3a98692b3503aef02f1d865942, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:16,511 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=104a1f3a98692b3503aef02f1d865942, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:55:16,511 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34681] ipc.CallRunner(144): callId: 170 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:41180 deadline: 1681527376505, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=104a1f3a98692b3503aef02f1d865942, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:16,690 INFO [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=206 (bloomFilter=true), to=hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/104a1f3a98692b3503aef02f1d865942/.tmp/B/8a147cdb74ac49999df355b4aba62ddb 2023-04-15 02:55:16,815 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=104a1f3a98692b3503aef02f1d865942, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:55:16,815 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34681] ipc.CallRunner(144): callId: 67 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41118 deadline: 1681527376810, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=104a1f3a98692b3503aef02f1d865942, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:16,815 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=104a1f3a98692b3503aef02f1d865942, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:55:16,815 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34681] ipc.CallRunner(144): callId: 172 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:41180 deadline: 1681527376814, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=104a1f3a98692b3503aef02f1d865942, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:16,816 DEBUG [Thread-2173] client.RpcRetryingCallerImpl(129): Call exception, tries=7, retries=16, started=8179 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=104a1f3a98692b3503aef02f1d865942, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_0' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1681527301940.104a1f3a98692b3503aef02f1d865942., hostname=jenkins-hbase20.apache.org,34681,1681527209133, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=104a1f3a98692b3503aef02f1d865942, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor31.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=104a1f3a98692b3503aef02f1d865942, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-15 02:55:16,825 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=104a1f3a98692b3503aef02f1d865942, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:55:16,825 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34681] ipc.CallRunner(144): callId: 73 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41156 deadline: 1681527376820, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=104a1f3a98692b3503aef02f1d865942, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:16,825 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=104a1f3a98692b3503aef02f1d865942, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:55:16,825 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34681] ipc.CallRunner(144): callId: 69 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41164 deadline: 1681527376821, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=104a1f3a98692b3503aef02f1d865942, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:16,827 DEBUG [Thread-2171] client.RpcRetryingCallerImpl(129): Call exception, tries=7, retries=16, started=8184 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=104a1f3a98692b3503aef02f1d865942, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_1' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1681527301940.104a1f3a98692b3503aef02f1d865942., hostname=jenkins-hbase20.apache.org,34681,1681527209133, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=104a1f3a98692b3503aef02f1d865942, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor31.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=104a1f3a98692b3503aef02f1d865942, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-15 02:55:16,827 DEBUG [Thread-2177] client.RpcRetryingCallerImpl(129): Call exception, tries=7, retries=16, started=8186 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=104a1f3a98692b3503aef02f1d865942, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_1' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1681527301940.104a1f3a98692b3503aef02f1d865942., hostname=jenkins-hbase20.apache.org,34681,1681527209133, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=104a1f3a98692b3503aef02f1d865942, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor31.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=104a1f3a98692b3503aef02f1d865942, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-15 02:55:17,099 INFO [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=206 (bloomFilter=true), to=hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/104a1f3a98692b3503aef02f1d865942/.tmp/C/75d1f54082ba4e1db8a180ae08f01d0b 2023-04-15 02:55:17,102 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/104a1f3a98692b3503aef02f1d865942/.tmp/A/ec5f7a40bd284b86a7878a2aaef40b61 as hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/104a1f3a98692b3503aef02f1d865942/A/ec5f7a40bd284b86a7878a2aaef40b61 2023-04-15 02:55:17,105 INFO [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/104a1f3a98692b3503aef02f1d865942/A/ec5f7a40bd284b86a7878a2aaef40b61, entries=150, sequenceid=206, filesize=25.3 K 2023-04-15 02:55:17,106 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/104a1f3a98692b3503aef02f1d865942/.tmp/B/8a147cdb74ac49999df355b4aba62ddb as hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/104a1f3a98692b3503aef02f1d865942/B/8a147cdb74ac49999df355b4aba62ddb 2023-04-15 02:55:17,109 INFO [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/104a1f3a98692b3503aef02f1d865942/B/8a147cdb74ac49999df355b4aba62ddb, entries=150, sequenceid=206, filesize=11.8 K 2023-04-15 02:55:17,109 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/104a1f3a98692b3503aef02f1d865942/.tmp/C/75d1f54082ba4e1db8a180ae08f01d0b as hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/104a1f3a98692b3503aef02f1d865942/C/75d1f54082ba4e1db8a180ae08f01d0b 2023-04-15 02:55:17,112 INFO [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/104a1f3a98692b3503aef02f1d865942/C/75d1f54082ba4e1db8a180ae08f01d0b, entries=150, sequenceid=206, filesize=11.8 K 2023-04-15 02:55:17,113 INFO [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~60.38 KB/61830, heapSize ~180 KB/184320, currentSize=114.05 KB/116790 for 104a1f3a98692b3503aef02f1d865942 in 1657ms, sequenceid=206, compaction requested=true 2023-04-15 02:55:17,113 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 104a1f3a98692b3503aef02f1d865942: 2023-04-15 02:55:17,113 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681527301940.104a1f3a98692b3503aef02f1d865942. 2023-04-15 02:55:17,113 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-15 02:55:17,113 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-15 02:55:17,113 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:17,114 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-15 02:55:17,114 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34681,1681527209133' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-15 02:55:17,118 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:17,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:17,118 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:17,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:55:17,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:55:17,119 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:17,119 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-15 02:55:17,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:55:17,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:55:17,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:55:17,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:17,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:55:17,120 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34681,1681527209133' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-15 02:55:17,120 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3952054c[Count = 0] remaining members to acquire global barrier 2023-04-15 02:55:17,120 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-15 02:55:17,120 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:17,123 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:17,123 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:17,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:17,123 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-15 02:55:17,123 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-15 02:55:17,123 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34681,1681527209133' in zk 2023-04-15 02:55:17,123 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:17,123 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-15 02:55:17,124 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-15 02:55:17,124 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:17,124 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-15 02:55:17,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:17,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:55:17,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:55:17,124 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-15 02:55:17,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:55:17,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:55:17,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:55:17,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:17,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:55:17,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:55:17,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:17,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34681,1681527209133': 2023-04-15 02:55:17,126 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34681,1681527209133' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-15 02:55:17,126 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-15 02:55:17,126 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-15 02:55:17,126 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-15 02:55:17,126 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-15 02:55:17,126 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-15 02:55:17,127 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:17,127 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-15 02:55:17,127 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:17,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:17,127 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-15 02:55:17,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-15 02:55:17,127 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:17,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:17,128 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:17,130 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-15 02:55:17,130 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-15 02:55:17,130 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:17,130 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-15 02:55:17,130 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-15 02:55:17,130 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-15 02:55:17,130 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-15 02:55:17,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-15 02:55:17,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:17,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:55:17,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:55:17,130 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:17,130 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:55:17,130 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:55:17,130 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:17,130 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:17,130 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:17,130 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:17,130 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-15 02:55:17,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-15 02:55:17,130 DEBUG [Thread-2181] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-15 02:55:17,131 DEBUG [Thread-2181] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-15 02:55:17,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:55:17,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:55:17,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:55:17,131 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-15 02:55:17,231 DEBUG [Thread-2181] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-15 02:55:17,231 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-15 02:55:17,232 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-15 02:55:17,233 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-15 02:55:17,233 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-15 02:55:17,233 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-15 02:55:17,233 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-15 02:55:17,233 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-15 02:55:17,233 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-15 02:55:17,234 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:17,234 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:55:17,234 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-15 02:55:17,234 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:17,234 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-15 02:55:17,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-15 02:55:17,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:55:17,235 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:17,235 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-15 02:55:17,235 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:17,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-15 02:55:17,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:55:17,235 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-15 02:55:17,235 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-15 02:55:17,235 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-15 02:55:17,235 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-15 02:55:17,235 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-15 02:55:17,235 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-15 02:55:17,235 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-15 02:55:17,235 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-15 02:55:17,235 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681527301940.104a1f3a98692b3503aef02f1d865942. 2023-04-15 02:55:17,236 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681527301940.104a1f3a98692b3503aef02f1d865942. started... 2023-04-15 02:55:17,236 INFO [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 104a1f3a98692b3503aef02f1d865942 3/3 column families, dataSize=114.05 KB heapSize=339.42 KB 2023-04-15 02:55:17,236 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 104a1f3a98692b3503aef02f1d865942, store=A 2023-04-15 02:55:17,236 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-15 02:55:17,236 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 104a1f3a98692b3503aef02f1d865942, store=B 2023-04-15 02:55:17,236 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-15 02:55:17,236 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 104a1f3a98692b3503aef02f1d865942, store=C 2023-04-15 02:55:17,236 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-15 02:55:17,326 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34681] regionserver.HRegion(9158): Flush requested on 104a1f3a98692b3503aef02f1d865942 2023-04-15 02:55:17,326 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681527301940.104a1f3a98692b3503aef02f1d865942. as already flushing 2023-04-15 02:55:17,388 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=104a1f3a98692b3503aef02f1d865942, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:55:17,388 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34681] ipc.CallRunner(144): callId: 183 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:41180 deadline: 1681527377380, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=104a1f3a98692b3503aef02f1d865942, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:17,493 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=104a1f3a98692b3503aef02f1d865942, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:55:17,493 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34681] ipc.CallRunner(144): callId: 185 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:41180 deadline: 1681527377489, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=104a1f3a98692b3503aef02f1d865942, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:17,646 INFO [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230415a97a6e901f3f41a19c66dde6ec889373 to hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230415a97a6e901f3f41a19c66dde6ec889373 2023-04-15 02:55:17,701 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=104a1f3a98692b3503aef02f1d865942, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:55:17,702 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34681] ipc.CallRunner(144): callId: 187 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:41180 deadline: 1681527377695, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=104a1f3a98692b3503aef02f1d865942, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:18,012 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=104a1f3a98692b3503aef02f1d865942, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:55:18,013 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34681] ipc.CallRunner(144): callId: 189 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:41180 deadline: 1681527378004, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=104a1f3a98692b3503aef02f1d865942, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:18,064 INFO [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=226, memsize=38.0 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/104a1f3a98692b3503aef02f1d865942/.tmp/A/fab12f4dfac74644915b7dd6c995f10d 2023-04-15 02:55:18,073 INFO [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=226 (bloomFilter=true), to=hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/104a1f3a98692b3503aef02f1d865942/.tmp/B/96e208c581044c5aaff7fe1378581a21 2023-04-15 02:55:18,093 INFO [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=226 (bloomFilter=true), to=hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/104a1f3a98692b3503aef02f1d865942/.tmp/C/44c32b452a754cc695579b554f2cddc9 2023-04-15 02:55:18,097 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/104a1f3a98692b3503aef02f1d865942/.tmp/A/fab12f4dfac74644915b7dd6c995f10d as hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/104a1f3a98692b3503aef02f1d865942/A/fab12f4dfac74644915b7dd6c995f10d 2023-04-15 02:55:18,101 INFO [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/104a1f3a98692b3503aef02f1d865942/A/fab12f4dfac74644915b7dd6c995f10d, entries=150, sequenceid=226, filesize=25.3 K 2023-04-15 02:55:18,102 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/104a1f3a98692b3503aef02f1d865942/.tmp/B/96e208c581044c5aaff7fe1378581a21 as hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/104a1f3a98692b3503aef02f1d865942/B/96e208c581044c5aaff7fe1378581a21 2023-04-15 02:55:18,108 INFO [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/104a1f3a98692b3503aef02f1d865942/B/96e208c581044c5aaff7fe1378581a21, entries=150, sequenceid=226, filesize=11.8 K 2023-04-15 02:55:18,109 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/104a1f3a98692b3503aef02f1d865942/.tmp/C/44c32b452a754cc695579b554f2cddc9 as hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/104a1f3a98692b3503aef02f1d865942/C/44c32b452a754cc695579b554f2cddc9 2023-04-15 02:55:18,116 INFO [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/104a1f3a98692b3503aef02f1d865942/C/44c32b452a754cc695579b554f2cddc9, entries=150, sequenceid=226, filesize=11.8 K 2023-04-15 02:55:18,116 INFO [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~114.05 KB/116790, heapSize ~339.38 KB/347520, currentSize=60.38 KB/61830 for 104a1f3a98692b3503aef02f1d865942 in 880ms, sequenceid=226, compaction requested=true 2023-04-15 02:55:18,117 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 104a1f3a98692b3503aef02f1d865942: 2023-04-15 02:55:18,117 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681527301940.104a1f3a98692b3503aef02f1d865942. 2023-04-15 02:55:18,117 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-15 02:55:18,117 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-15 02:55:18,117 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:18,120 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-15 02:55:18,120 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34681,1681527209133' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-15 02:55:18,121 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:18,121 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:18,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:18,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:55:18,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:55:18,122 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:18,122 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-15 02:55:18,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:55:18,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:55:18,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:55:18,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:18,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:55:18,123 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34681,1681527209133' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-15 02:55:18,123 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@702fc9ea[Count = 0] remaining members to acquire global barrier 2023-04-15 02:55:18,123 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-15 02:55:18,123 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:18,124 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:18,124 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:18,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:18,124 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-15 02:55:18,124 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-15 02:55:18,124 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34681,1681527209133' in zk 2023-04-15 02:55:18,124 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:18,124 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-15 02:55:18,125 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:18,125 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-15 02:55:18,125 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-15 02:55:18,125 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-15 02:55:18,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:18,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:55:18,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:55:18,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:55:18,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:55:18,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:55:18,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:18,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:55:18,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:55:18,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:18,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34681,1681527209133': 2023-04-15 02:55:18,128 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34681,1681527209133' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-15 02:55:18,128 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-15 02:55:18,128 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-15 02:55:18,128 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-15 02:55:18,128 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-15 02:55:18,128 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-15 02:55:18,129 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:18,129 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:18,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:18,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:55:18,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:55:18,129 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:18,129 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-15 02:55:18,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:18,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:55:18,130 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-15 02:55:18,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-15 02:55:18,130 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:18,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:55:18,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:18,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:55:18,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:55:18,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:18,131 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:18,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:55:18,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:55:18,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:18,133 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-15 02:55:18,133 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:18,133 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-15 02:55:18,133 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:55:18,133 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-15 02:55:18,133 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-15 02:55:18,133 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-15 02:55:18,133 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:55:18,134 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:18,134 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:18,134 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:18,134 DEBUG [Thread-2181] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-15 02:55:18,134 DEBUG [Thread-2181] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-15 02:55:18,133 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-15 02:55:18,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-15 02:55:18,133 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-15 02:55:18,134 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:18,134 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-15 02:55:18,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-15 02:55:18,234 DEBUG [Thread-2181] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-15 02:55:18,234 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-15 02:55:18,236 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-15 02:55:18,237 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-15 02:55:18,237 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-15 02:55:18,237 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-15 02:55:18,237 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-15 02:55:18,237 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-15 02:55:18,237 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-15 02:55:18,237 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:18,237 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:55:18,238 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-15 02:55:18,238 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-15 02:55:18,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-15 02:55:18,238 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:18,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:55:18,239 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:18,239 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-15 02:55:18,239 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:18,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-15 02:55:18,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:55:18,239 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-15 02:55:18,239 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-15 02:55:18,239 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-15 02:55:18,239 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-15 02:55:18,239 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-15 02:55:18,240 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-15 02:55:18,240 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-15 02:55:18,240 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-15 02:55:18,240 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681527301940.104a1f3a98692b3503aef02f1d865942. 2023-04-15 02:55:18,240 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681527301940.104a1f3a98692b3503aef02f1d865942. started... 2023-04-15 02:55:18,240 INFO [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 104a1f3a98692b3503aef02f1d865942 3/3 column families, dataSize=60.38 KB heapSize=180.05 KB 2023-04-15 02:55:18,240 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 104a1f3a98692b3503aef02f1d865942, store=A 2023-04-15 02:55:18,240 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-15 02:55:18,240 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 104a1f3a98692b3503aef02f1d865942, store=B 2023-04-15 02:55:18,240 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-15 02:55:18,240 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 104a1f3a98692b3503aef02f1d865942, store=C 2023-04-15 02:55:18,240 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-15 02:55:18,539 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34681] regionserver.HRegion(9158): Flush requested on 104a1f3a98692b3503aef02f1d865942 2023-04-15 02:55:18,539 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681527301940.104a1f3a98692b3503aef02f1d865942. as already flushing 2023-04-15 02:55:18,657 INFO [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304156c84924d224848c8aa54ce25eefcac07 to hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304156c84924d224848c8aa54ce25eefcac07 2023-04-15 02:55:18,672 INFO [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=238, memsize=20.1 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/104a1f3a98692b3503aef02f1d865942/.tmp/A/25853bcb93364adf8c9bed5374757bf0 2023-04-15 02:55:18,687 INFO [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=238 (bloomFilter=true), to=hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/104a1f3a98692b3503aef02f1d865942/.tmp/B/6590adc88db04832ad8a64a343777501 2023-04-15 02:55:18,834 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=104a1f3a98692b3503aef02f1d865942, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:55:18,834 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34681] ipc.CallRunner(144): callId: 208 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:41180 deadline: 1681527378827, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=104a1f3a98692b3503aef02f1d865942, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:18,943 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=104a1f3a98692b3503aef02f1d865942, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:55:18,943 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34681] ipc.CallRunner(144): callId: 210 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:41180 deadline: 1681527378935, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=104a1f3a98692b3503aef02f1d865942, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:19,104 INFO [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=238 (bloomFilter=true), to=hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/104a1f3a98692b3503aef02f1d865942/.tmp/C/dc2f567be0b342b3b6bf7a4e71125802 2023-04-15 02:55:19,107 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/104a1f3a98692b3503aef02f1d865942/.tmp/A/25853bcb93364adf8c9bed5374757bf0 as hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/104a1f3a98692b3503aef02f1d865942/A/25853bcb93364adf8c9bed5374757bf0 2023-04-15 02:55:19,110 INFO [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/104a1f3a98692b3503aef02f1d865942/A/25853bcb93364adf8c9bed5374757bf0, entries=100, sequenceid=238, filesize=18.5 K 2023-04-15 02:55:19,111 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/104a1f3a98692b3503aef02f1d865942/.tmp/B/6590adc88db04832ad8a64a343777501 as hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/104a1f3a98692b3503aef02f1d865942/B/6590adc88db04832ad8a64a343777501 2023-04-15 02:55:19,113 INFO [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/104a1f3a98692b3503aef02f1d865942/B/6590adc88db04832ad8a64a343777501, entries=100, sequenceid=238, filesize=9.4 K 2023-04-15 02:55:19,114 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/104a1f3a98692b3503aef02f1d865942/.tmp/C/dc2f567be0b342b3b6bf7a4e71125802 as hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/104a1f3a98692b3503aef02f1d865942/C/dc2f567be0b342b3b6bf7a4e71125802 2023-04-15 02:55:19,117 INFO [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/104a1f3a98692b3503aef02f1d865942/C/dc2f567be0b342b3b6bf7a4e71125802, entries=100, sequenceid=238, filesize=9.4 K 2023-04-15 02:55:19,117 INFO [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~60.38 KB/61830, heapSize ~180 KB/184320, currentSize=114.05 KB/116790 for 104a1f3a98692b3503aef02f1d865942 in 877ms, sequenceid=238, compaction requested=true 2023-04-15 02:55:19,118 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 104a1f3a98692b3503aef02f1d865942: 2023-04-15 02:55:19,118 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681527301940.104a1f3a98692b3503aef02f1d865942. 2023-04-15 02:55:19,118 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-15 02:55:19,118 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-15 02:55:19,118 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:19,118 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-15 02:55:19,118 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34681,1681527209133' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-15 02:55:19,123 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:19,123 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:19,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:19,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:55:19,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:55:19,123 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:19,123 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-15 02:55:19,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:55:19,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:55:19,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:55:19,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:19,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:55:19,124 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34681,1681527209133' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-15 02:55:19,124 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@74ad4a85[Count = 0] remaining members to acquire global barrier 2023-04-15 02:55:19,124 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-15 02:55:19,124 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:19,125 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:19,125 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:19,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:19,125 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-15 02:55:19,125 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-15 02:55:19,125 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:19,125 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-15 02:55:19,125 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34681,1681527209133' in zk 2023-04-15 02:55:19,126 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-15 02:55:19,126 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:19,126 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-15 02:55:19,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:19,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:55:19,126 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-15 02:55:19,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:55:19,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:55:19,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:55:19,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:55:19,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:19,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:55:19,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:55:19,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:19,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34681,1681527209133': 2023-04-15 02:55:19,128 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34681,1681527209133' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-15 02:55:19,128 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-15 02:55:19,128 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-15 02:55:19,128 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-15 02:55:19,128 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-15 02:55:19,128 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-15 02:55:19,129 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:19,129 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:19,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:19,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:55:19,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:55:19,129 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-15 02:55:19,129 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:19,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:19,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:55:19,129 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:19,129 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-15 02:55:19,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-15 02:55:19,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:55:19,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:19,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:55:19,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:55:19,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:19,130 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:19,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:55:19,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:55:19,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:19,131 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:19,131 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-15 02:55:19,131 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:55:19,131 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-15 02:55:19,131 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-15 02:55:19,131 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-15 02:55:19,131 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-15 02:55:19,131 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-15 02:55:19,132 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-15 02:55:19,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-15 02:55:19,131 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:55:19,132 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:19,132 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:19,132 DEBUG [Thread-2181] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-15 02:55:19,132 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:19,132 DEBUG [Thread-2181] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-15 02:55:19,132 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:19,132 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-15 02:55:19,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-15 02:55:19,150 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34681] regionserver.HRegion(9158): Flush requested on 104a1f3a98692b3503aef02f1d865942 2023-04-15 02:55:19,150 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 104a1f3a98692b3503aef02f1d865942 3/3 column families, dataSize=120.76 KB heapSize=360.05 KB 2023-04-15 02:55:19,151 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 104a1f3a98692b3503aef02f1d865942, store=A 2023-04-15 02:55:19,151 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-15 02:55:19,151 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 104a1f3a98692b3503aef02f1d865942, store=B 2023-04-15 02:55:19,151 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-15 02:55:19,151 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 104a1f3a98692b3503aef02f1d865942, store=C 2023-04-15 02:55:19,151 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-15 02:55:19,165 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023041528c65241600346cab5b612fb8530642c to hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023041528c65241600346cab5b612fb8530642c 2023-04-15 02:55:19,232 DEBUG [Thread-2181] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-15 02:55:19,232 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-15 02:55:19,233 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-15 02:55:19,235 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-15 02:55:19,235 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-15 02:55:19,235 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-15 02:55:19,235 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-15 02:55:19,235 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-15 02:55:19,235 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-15 02:55:19,235 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:19,235 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:55:19,236 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:19,236 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-15 02:55:19,236 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-15 02:55:19,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-15 02:55:19,236 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:19,236 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-15 02:55:19,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:55:19,236 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:19,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-15 02:55:19,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:55:19,237 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-15 02:55:19,237 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-15 02:55:19,237 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-15 02:55:19,237 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-15 02:55:19,237 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-15 02:55:19,237 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-15 02:55:19,237 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-15 02:55:19,237 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-15 02:55:19,237 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681527301940.104a1f3a98692b3503aef02f1d865942. 2023-04-15 02:55:19,237 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681527301940.104a1f3a98692b3503aef02f1d865942. started... 2023-04-15 02:55:19,237 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681527301940.104a1f3a98692b3503aef02f1d865942. as already flushing 2023-04-15 02:55:19,237 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681527301940.104a1f3a98692b3503aef02f1d865942. 2023-04-15 02:55:19,237 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-15 02:55:19,237 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-15 02:55:19,237 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:19,237 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-15 02:55:19,237 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34681,1681527209133' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-15 02:55:19,238 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:19,238 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:19,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:19,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:55:19,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:55:19,238 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:19,238 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-15 02:55:19,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:55:19,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:55:19,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:55:19,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:19,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:55:19,240 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34681,1681527209133' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-15 02:55:19,240 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-15 02:55:19,240 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7cb18041[Count = 0] remaining members to acquire global barrier 2023-04-15 02:55:19,240 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:19,240 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:19,240 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:19,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:19,241 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-15 02:55:19,241 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-15 02:55:19,241 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34681,1681527209133' in zk 2023-04-15 02:55:19,241 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:19,241 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-15 02:55:19,241 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:19,241 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-15 02:55:19,241 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-15 02:55:19,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:19,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:55:19,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:55:19,241 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-15 02:55:19,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:55:19,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:55:19,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:55:19,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:19,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:55:19,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:55:19,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:19,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34681,1681527209133': 2023-04-15 02:55:19,243 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34681,1681527209133' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-15 02:55:19,243 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-15 02:55:19,243 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-15 02:55:19,243 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-15 02:55:19,243 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-15 02:55:19,243 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-15 02:55:19,244 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:19,244 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:19,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:19,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:55:19,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:55:19,244 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:19,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:19,244 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-15 02:55:19,244 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-15 02:55:19,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-15 02:55:19,244 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:19,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:55:19,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:19,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:55:19,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:55:19,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:55:19,245 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:19,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:19,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:55:19,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:55:19,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:19,246 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=104a1f3a98692b3503aef02f1d865942, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:55:19,246 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34681] ipc.CallRunner(144): callId: 221 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:41180 deadline: 1681527379239, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=104a1f3a98692b3503aef02f1d865942, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:19,247 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:19,247 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:55:19,247 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:55:19,247 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:19,247 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-15 02:55:19,247 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-15 02:55:19,247 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:19,247 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-15 02:55:19,247 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-15 02:55:19,247 DEBUG [Thread-2181] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-15 02:55:19,247 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:19,247 DEBUG [Thread-2181] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-15 02:55:19,247 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-15 02:55:19,247 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-15 02:55:19,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-15 02:55:19,247 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:19,247 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-15 02:55:19,247 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-15 02:55:19,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-15 02:55:19,347 DEBUG [Thread-2181] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-15 02:55:19,348 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-15 02:55:19,349 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-15 02:55:19,350 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-15 02:55:19,350 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-15 02:55:19,350 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-15 02:55:19,350 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-15 02:55:19,350 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-15 02:55:19,350 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-15 02:55:19,350 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:19,350 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:55:19,351 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-15 02:55:19,351 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:19,351 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-15 02:55:19,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-15 02:55:19,351 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:19,351 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-15 02:55:19,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:55:19,351 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:19,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-15 02:55:19,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:55:19,352 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-15 02:55:19,352 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-15 02:55:19,352 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-15 02:55:19,352 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-15 02:55:19,352 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-15 02:55:19,352 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-15 02:55:19,352 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-15 02:55:19,352 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-15 02:55:19,352 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681527301940.104a1f3a98692b3503aef02f1d865942. 2023-04-15 02:55:19,352 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681527301940.104a1f3a98692b3503aef02f1d865942. started... 2023-04-15 02:55:19,352 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681527301940.104a1f3a98692b3503aef02f1d865942. as already flushing 2023-04-15 02:55:19,352 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681527301940.104a1f3a98692b3503aef02f1d865942. 2023-04-15 02:55:19,352 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-15 02:55:19,352 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-15 02:55:19,352 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:19,352 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-15 02:55:19,352 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34681,1681527209133' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-15 02:55:19,353 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:19,353 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:19,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:19,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:55:19,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:55:19,353 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:19,353 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-15 02:55:19,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:55:19,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:55:19,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:55:19,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:19,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:55:19,354 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34681,1681527209133' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-15 02:55:19,355 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@55e092fe[Count = 0] remaining members to acquire global barrier 2023-04-15 02:55:19,355 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-15 02:55:19,355 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:19,355 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=104a1f3a98692b3503aef02f1d865942, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:55:19,355 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34681] ipc.CallRunner(144): callId: 223 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:41180 deadline: 1681527379347, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=104a1f3a98692b3503aef02f1d865942, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:19,356 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:19,356 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:19,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:19,356 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-15 02:55:19,356 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-15 02:55:19,356 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:19,356 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-15 02:55:19,356 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34681,1681527209133' in zk 2023-04-15 02:55:19,357 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:19,357 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-15 02:55:19,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:19,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:55:19,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:55:19,357 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-15 02:55:19,357 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-15 02:55:19,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:55:19,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:55:19,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:55:19,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:19,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:55:19,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:55:19,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:19,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34681,1681527209133': 2023-04-15 02:55:19,359 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34681,1681527209133' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-15 02:55:19,359 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-15 02:55:19,359 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-15 02:55:19,359 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-15 02:55:19,359 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-15 02:55:19,359 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-15 02:55:19,361 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:19,361 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:19,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:19,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:55:19,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:55:19,361 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:19,361 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-15 02:55:19,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:19,361 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:19,361 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-15 02:55:19,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-15 02:55:19,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:55:19,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:19,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:55:19,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:55:19,362 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:19,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:55:19,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:19,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:55:19,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:55:19,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:19,363 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-15 02:55:19,363 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:19,363 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-15 02:55:19,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-15 02:55:19,363 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-15 02:55:19,364 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-15 02:55:19,363 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:55:19,364 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-15 02:55:19,363 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-15 02:55:19,364 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-15 02:55:19,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-15 02:55:19,364 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:55:19,364 DEBUG [Thread-2181] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-15 02:55:19,364 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:19,364 DEBUG [Thread-2181] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-15 02:55:19,364 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:19,364 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-15 02:55:19,364 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:19,364 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:19,464 DEBUG [Thread-2181] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-15 02:55:19,465 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-15 02:55:19,466 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-15 02:55:19,467 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-15 02:55:19,467 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-15 02:55:19,467 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-15 02:55:19,467 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-15 02:55:19,467 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-15 02:55:19,467 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-15 02:55:19,467 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:19,467 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:55:19,468 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:19,468 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-15 02:55:19,468 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-15 02:55:19,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-15 02:55:19,468 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:19,468 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-15 02:55:19,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:55:19,469 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:19,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-15 02:55:19,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:55:19,469 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-15 02:55:19,469 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-15 02:55:19,469 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-15 02:55:19,469 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-15 02:55:19,469 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-15 02:55:19,469 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-15 02:55:19,470 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-15 02:55:19,470 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-15 02:55:19,470 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681527301940.104a1f3a98692b3503aef02f1d865942. 2023-04-15 02:55:19,470 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681527301940.104a1f3a98692b3503aef02f1d865942. started... 2023-04-15 02:55:19,470 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681527301940.104a1f3a98692b3503aef02f1d865942. as already flushing 2023-04-15 02:55:19,470 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681527301940.104a1f3a98692b3503aef02f1d865942. 2023-04-15 02:55:19,470 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-15 02:55:19,470 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-15 02:55:19,470 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:19,470 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-15 02:55:19,471 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34681,1681527209133' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-15 02:55:19,471 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:19,471 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:19,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:19,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:55:19,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:55:19,471 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:19,472 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-15 02:55:19,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:55:19,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:55:19,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:55:19,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:19,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:55:19,473 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34681,1681527209133' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-15 02:55:19,473 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-15 02:55:19,473 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5b0e6131[Count = 0] remaining members to acquire global barrier 2023-04-15 02:55:19,473 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:19,474 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:19,474 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:19,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:19,474 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-15 02:55:19,474 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-15 02:55:19,474 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:19,474 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-15 02:55:19,474 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34681,1681527209133' in zk 2023-04-15 02:55:19,475 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:19,475 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-15 02:55:19,475 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-15 02:55:19,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:19,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:55:19,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:55:19,475 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-15 02:55:19,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:55:19,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:55:19,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:55:19,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:19,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:55:19,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:55:19,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:19,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34681,1681527209133': 2023-04-15 02:55:19,476 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34681,1681527209133' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-15 02:55:19,476 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-15 02:55:19,476 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-15 02:55:19,476 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-15 02:55:19,476 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-15 02:55:19,476 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-15 02:55:19,477 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:19,477 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:19,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:19,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:55:19,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:55:19,477 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:19,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:19,477 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-15 02:55:19,478 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:19,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:55:19,478 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-15 02:55:19,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-15 02:55:19,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:55:19,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:19,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:55:19,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:55:19,478 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:19,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:19,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:55:19,488 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-15 02:55:19,488 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:19,488 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-15 02:55:19,488 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-15 02:55:19,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-15 02:55:19,488 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-15 02:55:19,488 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-15 02:55:19,488 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-15 02:55:19,488 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:55:19,488 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-15 02:55:19,488 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:55:19,488 DEBUG [Thread-2181] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-15 02:55:19,488 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:19,488 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-15 02:55:19,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-15 02:55:19,488 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:19,488 DEBUG [Thread-2181] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-15 02:55:19,488 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:19,488 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:19,563 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=104a1f3a98692b3503aef02f1d865942, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:55:19,563 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34681] ipc.CallRunner(144): callId: 225 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:41180 deadline: 1681527379557, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=104a1f3a98692b3503aef02f1d865942, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:19,575 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=259, memsize=40.3 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/104a1f3a98692b3503aef02f1d865942/.tmp/A/b9c13adc25ec4fe5b85c9fd9f11b5c24 2023-04-15 02:55:19,588 DEBUG [Thread-2181] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-15 02:55:19,590 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-15 02:55:19,591 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=40.25 KB at sequenceid=259 (bloomFilter=true), to=hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/104a1f3a98692b3503aef02f1d865942/.tmp/B/3551e48afafc4d49a049bfee2388e032 2023-04-15 02:55:19,591 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-15 02:55:19,592 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-15 02:55:19,592 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-15 02:55:19,593 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-15 02:55:19,593 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-15 02:55:19,593 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-15 02:55:19,593 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-15 02:55:19,593 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:19,593 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:55:19,594 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-15 02:55:19,594 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:19,594 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-15 02:55:19,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-15 02:55:19,595 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:19,595 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-15 02:55:19,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:55:19,595 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:19,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-15 02:55:19,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:55:19,595 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-15 02:55:19,595 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-15 02:55:19,595 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-15 02:55:19,596 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-15 02:55:19,596 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-15 02:55:19,596 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-15 02:55:19,597 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-15 02:55:19,597 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-15 02:55:19,597 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681527301940.104a1f3a98692b3503aef02f1d865942. 2023-04-15 02:55:19,597 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681527301940.104a1f3a98692b3503aef02f1d865942. started... 2023-04-15 02:55:19,597 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681527301940.104a1f3a98692b3503aef02f1d865942. as already flushing 2023-04-15 02:55:19,597 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681527301940.104a1f3a98692b3503aef02f1d865942. 2023-04-15 02:55:19,597 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-15 02:55:19,597 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-15 02:55:19,597 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:19,598 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-15 02:55:19,598 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34681,1681527209133' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-15 02:55:19,599 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:19,599 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:19,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:19,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:55:19,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:55:19,599 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:19,599 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-15 02:55:19,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:55:19,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:55:19,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:55:19,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:19,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:55:19,600 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34681,1681527209133' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-15 02:55:19,600 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@45e9e1b7[Count = 0] remaining members to acquire global barrier 2023-04-15 02:55:19,600 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-15 02:55:19,600 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:19,601 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:19,601 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:19,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:19,602 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-15 02:55:19,602 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-15 02:55:19,602 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34681,1681527209133' in zk 2023-04-15 02:55:19,602 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=40.25 KB at sequenceid=259 (bloomFilter=true), to=hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/104a1f3a98692b3503aef02f1d865942/.tmp/C/41866252f56145e09d7441da4e316361 2023-04-15 02:55:19,602 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:19,602 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-15 02:55:19,603 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:19,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:19,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:55:19,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:55:19,604 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-15 02:55:19,604 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-15 02:55:19,604 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-15 02:55:19,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:55:19,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:55:19,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:55:19,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:19,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:55:19,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:55:19,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:19,607 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/104a1f3a98692b3503aef02f1d865942/.tmp/A/b9c13adc25ec4fe5b85c9fd9f11b5c24 as hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/104a1f3a98692b3503aef02f1d865942/A/b9c13adc25ec4fe5b85c9fd9f11b5c24 2023-04-15 02:55:19,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34681,1681527209133': 2023-04-15 02:55:19,608 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34681,1681527209133' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-15 02:55:19,608 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-15 02:55:19,608 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-15 02:55:19,608 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-15 02:55:19,608 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-15 02:55:19,608 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-15 02:55:19,612 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/104a1f3a98692b3503aef02f1d865942/A/b9c13adc25ec4fe5b85c9fd9f11b5c24, entries=200, sequenceid=259, filesize=32.2 K 2023-04-15 02:55:19,612 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/104a1f3a98692b3503aef02f1d865942/.tmp/B/3551e48afafc4d49a049bfee2388e032 as hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/104a1f3a98692b3503aef02f1d865942/B/3551e48afafc4d49a049bfee2388e032 2023-04-15 02:55:19,614 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:19,614 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:19,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:19,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:55:19,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:55:19,614 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:19,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:19,614 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-15 02:55:19,615 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-15 02:55:19,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-15 02:55:19,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:55:19,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:19,615 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:19,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:55:19,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:55:19,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:55:19,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:19,616 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:19,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:55:19,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:55:19,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:19,617 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/104a1f3a98692b3503aef02f1d865942/B/3551e48afafc4d49a049bfee2388e032, entries=150, sequenceid=259, filesize=11.9 K 2023-04-15 02:55:19,618 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/104a1f3a98692b3503aef02f1d865942/.tmp/C/41866252f56145e09d7441da4e316361 as hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/104a1f3a98692b3503aef02f1d865942/C/41866252f56145e09d7441da4e316361 2023-04-15 02:55:19,621 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/104a1f3a98692b3503aef02f1d865942/C/41866252f56145e09d7441da4e316361, entries=150, sequenceid=259, filesize=11.9 K 2023-04-15 02:55:19,621 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-15 02:55:19,621 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:19,621 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-15 02:55:19,621 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-15 02:55:19,621 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-15 02:55:19,621 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:55:19,621 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-15 02:55:19,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-15 02:55:19,621 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-15 02:55:19,622 DEBUG [Thread-2181] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-15 02:55:19,621 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~120.76 KB/123660, heapSize ~360 KB/368640, currentSize=53.67 KB/54960 for 104a1f3a98692b3503aef02f1d865942 in 471ms, sequenceid=259, compaction requested=true 2023-04-15 02:55:19,621 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:55:19,622 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 104a1f3a98692b3503aef02f1d865942: 2023-04-15 02:55:19,622 DEBUG [Thread-2181] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-15 02:55:19,622 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:19,622 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-15 02:55:19,622 DEBUG [RS:0;jenkins-hbase20:34681-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 12 store files, 0 compacting, 12 eligible, 16 blocking 2023-04-15 02:55:19,622 DEBUG [RS:0;jenkins-hbase20:34681-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 12 store files, 0 compacting, 12 eligible, 16 blocking 2023-04-15 02:55:19,622 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:19,622 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:19,622 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:19,622 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-15 02:55:19,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-15 02:55:19,622 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-15 02:55:19,622 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-15 02:55:19,625 DEBUG [RS:0;jenkins-hbase20:34681-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 10 files of size 258502 starting at candidate #1 after considering 52 permutations with 52 in ratio 2023-04-15 02:55:19,625 DEBUG [RS:0;jenkins-hbase20:34681-shortCompactions-0] regionserver.HStore(1912): 104a1f3a98692b3503aef02f1d865942/A is initiating minor compaction 2023-04-15 02:55:19,625 INFO [RS:0;jenkins-hbase20:34681-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 104a1f3a98692b3503aef02f1d865942/A in TestAcidGuarantees,,1681527301940.104a1f3a98692b3503aef02f1d865942. 2023-04-15 02:55:19,625 INFO [RS:0;jenkins-hbase20:34681-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/104a1f3a98692b3503aef02f1d865942/A/991d654bc57544848c53ef515055bdc8, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/104a1f3a98692b3503aef02f1d865942/A/c91da5073e754463afa07efe85663f69, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/104a1f3a98692b3503aef02f1d865942/A/93140167335e47b4970764378c84ea19, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/104a1f3a98692b3503aef02f1d865942/A/3127ee9695b9478896a251390e068af3, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/104a1f3a98692b3503aef02f1d865942/A/6efb8c0a53064a5cbeec795d2cb8faf7, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/104a1f3a98692b3503aef02f1d865942/A/ee83bff49f4a41f08854c29cf00b177d, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/104a1f3a98692b3503aef02f1d865942/A/10794015f4a84f0eb0bd35ac69adbe9e, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/104a1f3a98692b3503aef02f1d865942/A/ec5f7a40bd284b86a7878a2aaef40b61, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/104a1f3a98692b3503aef02f1d865942/A/fab12f4dfac74644915b7dd6c995f10d, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/104a1f3a98692b3503aef02f1d865942/A/25853bcb93364adf8c9bed5374757bf0] into tmpdir=hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/104a1f3a98692b3503aef02f1d865942/.tmp, totalSize=252.4 K 2023-04-15 02:55:19,625 DEBUG [RS:0;jenkins-hbase20:34681-shortCompactions-0] compactions.Compactor(207): Compacting 991d654bc57544848c53ef515055bdc8, keycount=200, bloomtype=ROW, size=31.9 K, encoding=NONE, compression=NONE, seqNum=83 2023-04-15 02:55:19,625 DEBUG [RS:0;jenkins-hbase20:34681-shortCompactions-0] compactions.Compactor(207): Compacting c91da5073e754463afa07efe85663f69, keycount=150, bloomtype=ROW, size=25.1 K, encoding=NONE, compression=NONE, seqNum=97 2023-04-15 02:55:19,625 DEBUG [RS:0;jenkins-hbase20:34681-shortCompactions-0] compactions.Compactor(207): Compacting 93140167335e47b4970764378c84ea19, keycount=150, bloomtype=ROW, size=25.2 K, encoding=NONE, compression=NONE, seqNum=130 2023-04-15 02:55:19,625 DEBUG [RS:0;jenkins-hbase20:34681-shortCompactions-0] compactions.Compactor(207): Compacting 3127ee9695b9478896a251390e068af3, keycount=150, bloomtype=ROW, size=25.3 K, encoding=NONE, compression=NONE, seqNum=142 2023-04-15 02:55:19,625 DEBUG [RS:0;jenkins-hbase20:34681-shortCompactions-0] compactions.Compactor(207): Compacting 6efb8c0a53064a5cbeec795d2cb8faf7, keycount=150, bloomtype=ROW, size=25.3 K, encoding=NONE, compression=NONE, seqNum=162 2023-04-15 02:55:19,625 DEBUG [RS:0;jenkins-hbase20:34681-shortCompactions-0] compactions.Compactor(207): Compacting ee83bff49f4a41f08854c29cf00b177d, keycount=150, bloomtype=ROW, size=25.3 K, encoding=NONE, compression=NONE, seqNum=174 2023-04-15 02:55:19,625 DEBUG [RS:0;jenkins-hbase20:34681-shortCompactions-0] compactions.Compactor(207): Compacting 10794015f4a84f0eb0bd35ac69adbe9e, keycount=150, bloomtype=ROW, size=25.3 K, encoding=NONE, compression=NONE, seqNum=194 2023-04-15 02:55:19,625 DEBUG [RS:0;jenkins-hbase20:34681-shortCompactions-0] compactions.Compactor(207): Compacting ec5f7a40bd284b86a7878a2aaef40b61, keycount=150, bloomtype=ROW, size=25.3 K, encoding=NONE, compression=NONE, seqNum=206 2023-04-15 02:55:19,625 DEBUG [RS:0;jenkins-hbase20:34681-shortCompactions-0] compactions.Compactor(207): Compacting fab12f4dfac74644915b7dd6c995f10d, keycount=150, bloomtype=ROW, size=25.3 K, encoding=NONE, compression=NONE, seqNum=226 2023-04-15 02:55:19,625 DEBUG [RS:0;jenkins-hbase20:34681-shortCompactions-0] compactions.Compactor(207): Compacting 25853bcb93364adf8c9bed5374757bf0, keycount=100, bloomtype=ROW, size=18.5 K, encoding=NONE, compression=NONE, seqNum=238 2023-04-15 02:55:19,625 DEBUG [RS:0;jenkins-hbase20:34681-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 10 files of size 117566 starting at candidate #1 after considering 52 permutations with 52 in ratio 2023-04-15 02:55:19,626 DEBUG [RS:0;jenkins-hbase20:34681-longCompactions-0] regionserver.HStore(1912): 104a1f3a98692b3503aef02f1d865942/B is initiating minor compaction 2023-04-15 02:55:19,626 INFO [RS:0;jenkins-hbase20:34681-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 104a1f3a98692b3503aef02f1d865942/B in TestAcidGuarantees,,1681527301940.104a1f3a98692b3503aef02f1d865942. 2023-04-15 02:55:19,626 INFO [RS:0;jenkins-hbase20:34681-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/104a1f3a98692b3503aef02f1d865942/B/7d51e0aeb60340a3b6351c7a7e06866e, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/104a1f3a98692b3503aef02f1d865942/B/f5c1bacbe88140468f37d7341a61ca1d, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/104a1f3a98692b3503aef02f1d865942/B/8f11a364308f4fe0a469630c6f70f2a3, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/104a1f3a98692b3503aef02f1d865942/B/ed4f08d05dca469d86fa7767faadf370, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/104a1f3a98692b3503aef02f1d865942/B/aad59967db1a4dcd8841e8dca8885b2a, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/104a1f3a98692b3503aef02f1d865942/B/0b220438b5504a3b93e3041e42f9f358, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/104a1f3a98692b3503aef02f1d865942/B/f5361327d59449549ed8db22f4e18551, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/104a1f3a98692b3503aef02f1d865942/B/8a147cdb74ac49999df355b4aba62ddb, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/104a1f3a98692b3503aef02f1d865942/B/96e208c581044c5aaff7fe1378581a21, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/104a1f3a98692b3503aef02f1d865942/B/6590adc88db04832ad8a64a343777501] into tmpdir=hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/104a1f3a98692b3503aef02f1d865942/.tmp, totalSize=114.8 K 2023-04-15 02:55:19,626 DEBUG [RS:0;jenkins-hbase20:34681-longCompactions-0] compactions.Compactor(207): Compacting 7d51e0aeb60340a3b6351c7a7e06866e, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=83 2023-04-15 02:55:19,626 DEBUG [RS:0;jenkins-hbase20:34681-longCompactions-0] compactions.Compactor(207): Compacting f5c1bacbe88140468f37d7341a61ca1d, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=97 2023-04-15 02:55:19,626 DEBUG [RS:0;jenkins-hbase20:34681-longCompactions-0] compactions.Compactor(207): Compacting 8f11a364308f4fe0a469630c6f70f2a3, keycount=150, bloomtype=ROW, size=11.7 K, encoding=NONE, compression=NONE, seqNum=130 2023-04-15 02:55:19,626 DEBUG [RS:0;jenkins-hbase20:34681-longCompactions-0] compactions.Compactor(207): Compacting ed4f08d05dca469d86fa7767faadf370, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=142 2023-04-15 02:55:19,626 DEBUG [RS:0;jenkins-hbase20:34681-longCompactions-0] compactions.Compactor(207): Compacting aad59967db1a4dcd8841e8dca8885b2a, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=162 2023-04-15 02:55:19,626 DEBUG [RS:0;jenkins-hbase20:34681-longCompactions-0] compactions.Compactor(207): Compacting 0b220438b5504a3b93e3041e42f9f358, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=174 2023-04-15 02:55:19,626 DEBUG [RS:0;jenkins-hbase20:34681-longCompactions-0] compactions.Compactor(207): Compacting f5361327d59449549ed8db22f4e18551, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=194 2023-04-15 02:55:19,626 DEBUG [RS:0;jenkins-hbase20:34681-longCompactions-0] compactions.Compactor(207): Compacting 8a147cdb74ac49999df355b4aba62ddb, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=206 2023-04-15 02:55:19,626 DEBUG [RS:0;jenkins-hbase20:34681-longCompactions-0] compactions.Compactor(207): Compacting 96e208c581044c5aaff7fe1378581a21, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=226 2023-04-15 02:55:19,626 DEBUG [RS:0;jenkins-hbase20:34681-longCompactions-0] compactions.Compactor(207): Compacting 6590adc88db04832ad8a64a343777501, keycount=100, bloomtype=ROW, size=9.4 K, encoding=NONE, compression=NONE, seqNum=238 2023-04-15 02:55:19,654 INFO [RS:0;jenkins-hbase20:34681-shortCompactions-0] throttle.PressureAwareThroughputController(145): 104a1f3a98692b3503aef02f1d865942#A#compaction#320 average throughput is 2.82 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-15 02:55:19,654 INFO [RS:0;jenkins-hbase20:34681-longCompactions-0] throttle.PressureAwareThroughputController(145): 104a1f3a98692b3503aef02f1d865942#B#compaction#321 average throughput is 1.64 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-15 02:55:19,722 DEBUG [Thread-2181] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-15 02:55:19,722 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-15 02:55:19,723 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-15 02:55:19,724 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-15 02:55:19,724 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-15 02:55:19,724 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-15 02:55:19,724 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-15 02:55:19,724 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-15 02:55:19,724 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-15 02:55:19,725 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:19,725 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:55:19,726 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:19,726 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-15 02:55:19,726 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-15 02:55:19,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-15 02:55:19,726 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:19,726 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-15 02:55:19,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:55:19,726 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:19,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-15 02:55:19,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:55:19,726 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-15 02:55:19,727 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-15 02:55:19,727 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-15 02:55:19,727 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-15 02:55:19,727 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-15 02:55:19,727 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-15 02:55:19,728 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-15 02:55:19,728 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-15 02:55:19,728 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681527301940.104a1f3a98692b3503aef02f1d865942. 2023-04-15 02:55:19,728 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681527301940.104a1f3a98692b3503aef02f1d865942. started... 2023-04-15 02:55:19,728 INFO [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 104a1f3a98692b3503aef02f1d865942 3/3 column families, dataSize=53.67 KB heapSize=160.13 KB 2023-04-15 02:55:19,729 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 104a1f3a98692b3503aef02f1d865942, store=A 2023-04-15 02:55:19,729 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-15 02:55:19,729 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 104a1f3a98692b3503aef02f1d865942, store=B 2023-04-15 02:55:19,729 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-15 02:55:19,729 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 104a1f3a98692b3503aef02f1d865942, store=C 2023-04-15 02:55:19,729 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-15 02:55:19,874 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34681] regionserver.HRegion(9158): Flush requested on 104a1f3a98692b3503aef02f1d865942 2023-04-15 02:55:19,874 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681527301940.104a1f3a98692b3503aef02f1d865942. as already flushing 2023-04-15 02:55:20,028 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681527319990, max timestamp=1681527320026Num uniques 859; 2023-04-15 02:55:20,028 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681527319990, max timestamp=1681527320026Num uniques 859; 2023-04-15 02:55:20,028 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-3] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-15 02:55:20,028 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681527319990, max timestamp=1681527320026Num uniques 859; 2023-04-15 02:55:20,028 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-4] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-15 02:55:20,028 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-8] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-15 02:55:20,064 DEBUG [RS:0;jenkins-hbase20:34681-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/104a1f3a98692b3503aef02f1d865942/.tmp/B/8b351a9d1f4740429641269e5aaf1ba4 as hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/104a1f3a98692b3503aef02f1d865942/B/8b351a9d1f4740429641269e5aaf1ba4 2023-04-15 02:55:20,068 INFO [RS:0;jenkins-hbase20:34681-longCompactions-0] regionserver.HStore(1652): Completed compaction of 10 file(s) in 104a1f3a98692b3503aef02f1d865942/B of 104a1f3a98692b3503aef02f1d865942 into 8b351a9d1f4740429641269e5aaf1ba4(size=12.1 K), total size for store is 35.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-15 02:55:20,068 DEBUG [RS:0;jenkins-hbase20:34681-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 104a1f3a98692b3503aef02f1d865942: 2023-04-15 02:55:20,068 INFO [RS:0;jenkins-hbase20:34681-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681527301940.104a1f3a98692b3503aef02f1d865942., storeName=104a1f3a98692b3503aef02f1d865942/B, priority=4, startTime=1681527319622; duration=0sec 2023-04-15 02:55:20,068 DEBUG [RS:0;jenkins-hbase20:34681-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-15 02:55:20,068 DEBUG [RS:0;jenkins-hbase20:34681-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 12 store files, 0 compacting, 12 eligible, 16 blocking 2023-04-15 02:55:20,071 DEBUG [RS:0;jenkins-hbase20:34681-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 10 files of size 117566 starting at candidate #1 after considering 52 permutations with 52 in ratio 2023-04-15 02:55:20,071 DEBUG [RS:0;jenkins-hbase20:34681-longCompactions-0] regionserver.HStore(1912): 104a1f3a98692b3503aef02f1d865942/C is initiating minor compaction 2023-04-15 02:55:20,071 INFO [RS:0;jenkins-hbase20:34681-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 104a1f3a98692b3503aef02f1d865942/C in TestAcidGuarantees,,1681527301940.104a1f3a98692b3503aef02f1d865942. 2023-04-15 02:55:20,071 INFO [RS:0;jenkins-hbase20:34681-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/104a1f3a98692b3503aef02f1d865942/C/520d53b28c344c838f7c4769d039de8c, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/104a1f3a98692b3503aef02f1d865942/C/ad3099311aa949b68b3948d71019ee48, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/104a1f3a98692b3503aef02f1d865942/C/5cb02c2a67b645b8b76212926ac890f2, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/104a1f3a98692b3503aef02f1d865942/C/fa4a013e158245edab7ec80baac43f81, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/104a1f3a98692b3503aef02f1d865942/C/62fe15263bef44aea7b83db93c46c4f9, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/104a1f3a98692b3503aef02f1d865942/C/8cf30cb2449c4d4c915a6fed58dbb6cb, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/104a1f3a98692b3503aef02f1d865942/C/4502e62cc1574f8bb74a3859a770227c, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/104a1f3a98692b3503aef02f1d865942/C/75d1f54082ba4e1db8a180ae08f01d0b, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/104a1f3a98692b3503aef02f1d865942/C/44c32b452a754cc695579b554f2cddc9, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/104a1f3a98692b3503aef02f1d865942/C/dc2f567be0b342b3b6bf7a4e71125802] into tmpdir=hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/104a1f3a98692b3503aef02f1d865942/.tmp, totalSize=114.8 K 2023-04-15 02:55:20,071 DEBUG [RS:0;jenkins-hbase20:34681-longCompactions-0] compactions.Compactor(207): Compacting 520d53b28c344c838f7c4769d039de8c, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=83 2023-04-15 02:55:20,071 DEBUG [RS:0;jenkins-hbase20:34681-longCompactions-0] compactions.Compactor(207): Compacting ad3099311aa949b68b3948d71019ee48, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=97 2023-04-15 02:55:20,071 DEBUG [RS:0;jenkins-hbase20:34681-longCompactions-0] compactions.Compactor(207): Compacting 5cb02c2a67b645b8b76212926ac890f2, keycount=150, bloomtype=ROW, size=11.7 K, encoding=NONE, compression=NONE, seqNum=130 2023-04-15 02:55:20,071 DEBUG [RS:0;jenkins-hbase20:34681-longCompactions-0] compactions.Compactor(207): Compacting fa4a013e158245edab7ec80baac43f81, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=142 2023-04-15 02:55:20,071 DEBUG [RS:0;jenkins-hbase20:34681-longCompactions-0] compactions.Compactor(207): Compacting 62fe15263bef44aea7b83db93c46c4f9, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=162 2023-04-15 02:55:20,071 DEBUG [RS:0;jenkins-hbase20:34681-longCompactions-0] compactions.Compactor(207): Compacting 8cf30cb2449c4d4c915a6fed58dbb6cb, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=174 2023-04-15 02:55:20,071 DEBUG [RS:0;jenkins-hbase20:34681-longCompactions-0] compactions.Compactor(207): Compacting 4502e62cc1574f8bb74a3859a770227c, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=194 2023-04-15 02:55:20,071 DEBUG [RS:0;jenkins-hbase20:34681-longCompactions-0] compactions.Compactor(207): Compacting 75d1f54082ba4e1db8a180ae08f01d0b, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=206 2023-04-15 02:55:20,071 DEBUG [RS:0;jenkins-hbase20:34681-longCompactions-0] compactions.Compactor(207): Compacting 44c32b452a754cc695579b554f2cddc9, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=226 2023-04-15 02:55:20,072 DEBUG [RS:0;jenkins-hbase20:34681-longCompactions-0] compactions.Compactor(207): Compacting dc2f567be0b342b3b6bf7a4e71125802, keycount=100, bloomtype=ROW, size=9.4 K, encoding=NONE, compression=NONE, seqNum=238 2023-04-15 02:55:20,084 INFO [RS:0;jenkins-hbase20:34681-longCompactions-0] throttle.PressureAwareThroughputController(145): 104a1f3a98692b3503aef02f1d865942#C#compaction#323 average throughput is 1.64 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-15 02:55:20,101 DEBUG [RS:0;jenkins-hbase20:34681-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/104a1f3a98692b3503aef02f1d865942/.tmp/C/486f243a5a8a47bfb55d6c75e878658d as hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/104a1f3a98692b3503aef02f1d865942/C/486f243a5a8a47bfb55d6c75e878658d 2023-04-15 02:55:20,105 INFO [RS:0;jenkins-hbase20:34681-longCompactions-0] regionserver.HStore(1652): Completed compaction of 10 file(s) in 104a1f3a98692b3503aef02f1d865942/C of 104a1f3a98692b3503aef02f1d865942 into 486f243a5a8a47bfb55d6c75e878658d(size=12.1 K), total size for store is 35.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-15 02:55:20,105 DEBUG [RS:0;jenkins-hbase20:34681-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 104a1f3a98692b3503aef02f1d865942: 2023-04-15 02:55:20,105 INFO [RS:0;jenkins-hbase20:34681-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681527301940.104a1f3a98692b3503aef02f1d865942., storeName=104a1f3a98692b3503aef02f1d865942/C, priority=4, startTime=1681527319622; duration=0sec 2023-04-15 02:55:20,105 DEBUG [RS:0;jenkins-hbase20:34681-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-15 02:55:20,146 INFO [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230415701eb9f89293494eb7b8de4a88717a22 to hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230415701eb9f89293494eb7b8de4a88717a22 2023-04-15 02:55:20,150 INFO [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=270, memsize=17.9 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/104a1f3a98692b3503aef02f1d865942/.tmp/A/5b535cd666c047c490051e7d46a2e5b2 2023-04-15 02:55:20,203 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=104a1f3a98692b3503aef02f1d865942, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:55:20,203 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34681] ipc.CallRunner(144): callId: 260 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:41180 deadline: 1681527380193, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=104a1f3a98692b3503aef02f1d865942, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:20,316 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=104a1f3a98692b3503aef02f1d865942, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:55:20,316 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34681] ipc.CallRunner(144): callId: 262 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:41180 deadline: 1681527380304, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=104a1f3a98692b3503aef02f1d865942, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:20,468 DEBUG [RS:0;jenkins-hbase20:34681-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/104a1f3a98692b3503aef02f1d865942/.tmp/A/33eb33ab038945ffaaa7258b29a920e3 as hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/104a1f3a98692b3503aef02f1d865942/A/33eb33ab038945ffaaa7258b29a920e3 2023-04-15 02:55:20,471 INFO [RS:0;jenkins-hbase20:34681-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 10 file(s) in 104a1f3a98692b3503aef02f1d865942/A of 104a1f3a98692b3503aef02f1d865942 into 33eb33ab038945ffaaa7258b29a920e3(size=25.6 K), total size for store is 83.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-15 02:55:20,471 DEBUG [RS:0;jenkins-hbase20:34681-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 104a1f3a98692b3503aef02f1d865942: 2023-04-15 02:55:20,471 INFO [RS:0;jenkins-hbase20:34681-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681527301940.104a1f3a98692b3503aef02f1d865942., storeName=104a1f3a98692b3503aef02f1d865942/A, priority=4, startTime=1681527319622; duration=0sec 2023-04-15 02:55:20,471 DEBUG [RS:0;jenkins-hbase20:34681-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-15 02:55:20,523 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=104a1f3a98692b3503aef02f1d865942, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:55:20,523 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34681] ipc.CallRunner(144): callId: 264 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:41180 deadline: 1681527380518, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=104a1f3a98692b3503aef02f1d865942, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:20,568 INFO [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=270 (bloomFilter=true), to=hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/104a1f3a98692b3503aef02f1d865942/.tmp/B/545eb950cd234d8db5b32be8dfbb9a99 2023-04-15 02:55:20,829 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=104a1f3a98692b3503aef02f1d865942, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:55:20,830 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34681] ipc.CallRunner(144): callId: 266 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:41180 deadline: 1681527380825, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=104a1f3a98692b3503aef02f1d865942, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:20,980 INFO [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=270 (bloomFilter=true), to=hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/104a1f3a98692b3503aef02f1d865942/.tmp/C/0ce75ec5e15543de8052b6a6b4e68ce1 2023-04-15 02:55:20,983 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/104a1f3a98692b3503aef02f1d865942/.tmp/A/5b535cd666c047c490051e7d46a2e5b2 as hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/104a1f3a98692b3503aef02f1d865942/A/5b535cd666c047c490051e7d46a2e5b2 2023-04-15 02:55:20,986 INFO [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/104a1f3a98692b3503aef02f1d865942/A/5b535cd666c047c490051e7d46a2e5b2, entries=150, sequenceid=270, filesize=25.4 K 2023-04-15 02:55:20,987 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/104a1f3a98692b3503aef02f1d865942/.tmp/B/545eb950cd234d8db5b32be8dfbb9a99 as hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/104a1f3a98692b3503aef02f1d865942/B/545eb950cd234d8db5b32be8dfbb9a99 2023-04-15 02:55:20,990 INFO [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/104a1f3a98692b3503aef02f1d865942/B/545eb950cd234d8db5b32be8dfbb9a99, entries=150, sequenceid=270, filesize=11.9 K 2023-04-15 02:55:20,991 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/104a1f3a98692b3503aef02f1d865942/.tmp/C/0ce75ec5e15543de8052b6a6b4e68ce1 as hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/104a1f3a98692b3503aef02f1d865942/C/0ce75ec5e15543de8052b6a6b4e68ce1 2023-04-15 02:55:21,000 INFO [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/104a1f3a98692b3503aef02f1d865942/C/0ce75ec5e15543de8052b6a6b4e68ce1, entries=150, sequenceid=270, filesize=11.9 K 2023-04-15 02:55:21,001 INFO [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=127.47 KB/130530 for 104a1f3a98692b3503aef02f1d865942 in 1273ms, sequenceid=270, compaction requested=true 2023-04-15 02:55:21,001 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 104a1f3a98692b3503aef02f1d865942: 2023-04-15 02:55:21,001 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681527301940.104a1f3a98692b3503aef02f1d865942. 2023-04-15 02:55:21,001 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-15 02:55:21,001 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-15 02:55:21,001 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:21,001 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-15 02:55:21,001 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34681,1681527209133' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-15 02:55:21,002 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:21,002 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:21,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:21,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:55:21,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:55:21,002 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:21,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:55:21,003 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-15 02:55:21,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:55:21,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:55:21,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:21,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:55:21,004 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34681,1681527209133' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-15 02:55:21,004 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@35aac2c3[Count = 0] remaining members to acquire global barrier 2023-04-15 02:55:21,004 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-15 02:55:21,004 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:21,004 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:21,004 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:21,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:21,004 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-15 02:55:21,004 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-15 02:55:21,004 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34681,1681527209133' in zk 2023-04-15 02:55:21,005 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:21,005 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-15 02:55:21,006 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-15 02:55:21,006 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-15 02:55:21,006 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:21,006 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-15 02:55:21,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:21,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:55:21,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:55:21,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:55:21,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:55:21,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:55:21,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:21,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:55:21,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:55:21,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:21,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34681,1681527209133': 2023-04-15 02:55:21,008 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34681,1681527209133' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-15 02:55:21,008 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-15 02:55:21,008 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-15 02:55:21,008 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-15 02:55:21,008 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-15 02:55:21,008 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-15 02:55:21,009 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:21,009 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:21,009 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-15 02:55:21,009 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:21,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:21,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:21,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:55:21,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:55:21,010 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-15 02:55:21,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-15 02:55:21,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:55:21,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:21,010 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:21,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:55:21,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:55:21,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:55:21,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:21,011 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:21,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:55:21,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:55:21,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:21,013 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-15 02:55:21,013 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:21,013 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-15 02:55:21,013 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-15 02:55:21,013 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-15 02:55:21,013 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-15 02:55:21,013 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-15 02:55:21,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-15 02:55:21,013 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:55:21,013 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:55:21,013 DEBUG [Thread-2181] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-15 02:55:21,013 DEBUG [Thread-2181] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-15 02:55:21,013 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:21,013 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-15 02:55:21,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-15 02:55:21,013 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:21,013 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:21,013 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:21,113 DEBUG [Thread-2181] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-15 02:55:21,114 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-15 02:55:21,115 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-15 02:55:21,116 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-15 02:55:21,116 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-15 02:55:21,116 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-15 02:55:21,116 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-15 02:55:21,116 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-15 02:55:21,116 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-15 02:55:21,116 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:21,116 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:55:21,117 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:21,117 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-15 02:55:21,117 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-15 02:55:21,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-15 02:55:21,117 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:21,117 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-15 02:55:21,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:55:21,118 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:21,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-15 02:55:21,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:55:21,118 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-15 02:55:21,118 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-15 02:55:21,118 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-15 02:55:21,118 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-15 02:55:21,118 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-15 02:55:21,118 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-15 02:55:21,118 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-15 02:55:21,118 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-15 02:55:21,118 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681527301940.104a1f3a98692b3503aef02f1d865942. 2023-04-15 02:55:21,119 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681527301940.104a1f3a98692b3503aef02f1d865942. started... 2023-04-15 02:55:21,119 INFO [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 104a1f3a98692b3503aef02f1d865942 3/3 column families, dataSize=127.47 KB heapSize=365.06 KB 2023-04-15 02:55:21,119 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 104a1f3a98692b3503aef02f1d865942, store=A 2023-04-15 02:55:21,119 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-15 02:55:21,119 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 104a1f3a98692b3503aef02f1d865942, store=B 2023-04-15 02:55:21,119 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-15 02:55:21,119 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 104a1f3a98692b3503aef02f1d865942, store=C 2023-04-15 02:55:21,119 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-15 02:55:21,339 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34681] regionserver.HRegion(9158): Flush requested on 104a1f3a98692b3503aef02f1d865942 2023-04-15 02:55:21,340 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681527301940.104a1f3a98692b3503aef02f1d865942. as already flushing 2023-04-15 02:55:21,400 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=104a1f3a98692b3503aef02f1d865942, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:55:21,400 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34681] ipc.CallRunner(144): callId: 276 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:41180 deadline: 1681527381393, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=104a1f3a98692b3503aef02f1d865942, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:21,505 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=104a1f3a98692b3503aef02f1d865942, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:55:21,505 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34681] ipc.CallRunner(144): callId: 278 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:41180 deadline: 1681527381501, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=104a1f3a98692b3503aef02f1d865942, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:21,534 INFO [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230415f1273c1235c54def90fde1a4dc4aade0 to hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230415f1273c1235c54def90fde1a4dc4aade0 2023-04-15 02:55:21,712 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=104a1f3a98692b3503aef02f1d865942, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:55:21,713 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34681] ipc.CallRunner(144): callId: 280 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:41180 deadline: 1681527381707, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=104a1f3a98692b3503aef02f1d865942, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:21,939 INFO [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=309, memsize=42.5 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/104a1f3a98692b3503aef02f1d865942/.tmp/A/f891948ce31f453d8d3475787916ed1c 2023-04-15 02:55:22,017 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=104a1f3a98692b3503aef02f1d865942, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:55:22,017 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34681] ipc.CallRunner(144): callId: 282 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:41180 deadline: 1681527382013, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=104a1f3a98692b3503aef02f1d865942, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:22,350 INFO [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=309 (bloomFilter=true), to=hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/104a1f3a98692b3503aef02f1d865942/.tmp/B/343a248dec4643f4b11379095dd65595 2023-04-15 02:55:22,526 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=104a1f3a98692b3503aef02f1d865942, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:55:22,526 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34681] ipc.CallRunner(144): callId: 284 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:41180 deadline: 1681527382522, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=104a1f3a98692b3503aef02f1d865942, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:22,759 INFO [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=309 (bloomFilter=true), to=hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/104a1f3a98692b3503aef02f1d865942/.tmp/C/2005670e61c1453484cab603e00c404c 2023-04-15 02:55:22,763 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/104a1f3a98692b3503aef02f1d865942/.tmp/A/f891948ce31f453d8d3475787916ed1c as hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/104a1f3a98692b3503aef02f1d865942/A/f891948ce31f453d8d3475787916ed1c 2023-04-15 02:55:22,766 INFO [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/104a1f3a98692b3503aef02f1d865942/A/f891948ce31f453d8d3475787916ed1c, entries=150, sequenceid=309, filesize=25.4 K 2023-04-15 02:55:22,767 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/104a1f3a98692b3503aef02f1d865942/.tmp/B/343a248dec4643f4b11379095dd65595 as hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/104a1f3a98692b3503aef02f1d865942/B/343a248dec4643f4b11379095dd65595 2023-04-15 02:55:22,770 INFO [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/104a1f3a98692b3503aef02f1d865942/B/343a248dec4643f4b11379095dd65595, entries=150, sequenceid=309, filesize=11.9 K 2023-04-15 02:55:22,771 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/104a1f3a98692b3503aef02f1d865942/.tmp/C/2005670e61c1453484cab603e00c404c as hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/104a1f3a98692b3503aef02f1d865942/C/2005670e61c1453484cab603e00c404c 2023-04-15 02:55:22,774 INFO [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/104a1f3a98692b3503aef02f1d865942/C/2005670e61c1453484cab603e00c404c, entries=150, sequenceid=309, filesize=11.9 K 2023-04-15 02:55:22,774 INFO [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~127.47 KB/130530, heapSize ~365.02 KB/373776, currentSize=53.67 KB/54960 for 104a1f3a98692b3503aef02f1d865942 in 1655ms, sequenceid=309, compaction requested=true 2023-04-15 02:55:22,774 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 104a1f3a98692b3503aef02f1d865942: 2023-04-15 02:55:22,774 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681527301940.104a1f3a98692b3503aef02f1d865942. 2023-04-15 02:55:22,775 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-15 02:55:22,775 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-15 02:55:22,775 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:22,775 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-15 02:55:22,775 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34681,1681527209133' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-15 02:55:22,776 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:22,776 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:22,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:22,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:55:22,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:55:22,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:55:22,776 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:22,776 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-15 02:55:22,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:55:22,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:55:22,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:22,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:55:22,778 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34681,1681527209133' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-15 02:55:22,778 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-15 02:55:22,778 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@648cd1ea[Count = 0] remaining members to acquire global barrier 2023-04-15 02:55:22,778 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:22,778 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:22,778 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:22,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:22,778 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-15 02:55:22,778 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-15 02:55:22,778 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:22,778 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-15 02:55:22,778 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34681,1681527209133' in zk 2023-04-15 02:55:22,779 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:22,779 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-15 02:55:22,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:22,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:55:22,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:55:22,779 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-15 02:55:22,779 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-15 02:55:22,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:55:22,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:55:22,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:55:22,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:22,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:55:22,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:55:22,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:22,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34681,1681527209133': 2023-04-15 02:55:22,782 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34681,1681527209133' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-15 02:55:22,782 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-15 02:55:22,782 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-15 02:55:22,782 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-15 02:55:22,782 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-15 02:55:22,782 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-15 02:55:22,783 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:22,783 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-15 02:55:22,783 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:22,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:22,783 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:22,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:22,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:55:22,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:55:22,783 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-15 02:55:22,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-15 02:55:22,784 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:22,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:55:22,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:22,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:55:22,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:55:22,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:55:22,784 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:22,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:22,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:55:22,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:55:22,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:22,787 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:22,787 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:55:22,787 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:55:22,787 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:22,787 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-15 02:55:22,787 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-15 02:55:22,787 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-15 02:55:22,787 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-15 02:55:22,787 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-15 02:55:22,787 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:22,787 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-15 02:55:22,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-15 02:55:22,787 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-15 02:55:22,787 DEBUG [Thread-2181] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-15 02:55:22,787 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:22,787 DEBUG [Thread-2181] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-15 02:55:22,787 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:22,787 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-15 02:55:22,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-15 02:55:22,887 DEBUG [Thread-2181] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-15 02:55:22,888 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-15 02:55:22,889 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-15 02:55:22,890 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-15 02:55:22,890 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-15 02:55:22,890 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-15 02:55:22,890 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-15 02:55:22,890 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-15 02:55:22,890 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-15 02:55:22,890 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:22,890 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:55:22,891 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:22,891 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-15 02:55:22,891 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-15 02:55:22,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-15 02:55:22,891 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:22,891 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-15 02:55:22,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:55:22,893 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:22,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-15 02:55:22,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:55:22,893 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-15 02:55:22,893 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-15 02:55:22,893 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-15 02:55:22,893 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-15 02:55:22,893 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-15 02:55:22,893 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-15 02:55:22,893 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-15 02:55:22,893 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-15 02:55:22,893 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681527301940.104a1f3a98692b3503aef02f1d865942. 2023-04-15 02:55:22,894 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681527301940.104a1f3a98692b3503aef02f1d865942. started... 2023-04-15 02:55:22,894 INFO [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 104a1f3a98692b3503aef02f1d865942 3/3 column families, dataSize=53.67 KB heapSize=160.13 KB 2023-04-15 02:55:22,894 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 104a1f3a98692b3503aef02f1d865942, store=A 2023-04-15 02:55:22,894 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-15 02:55:22,894 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 104a1f3a98692b3503aef02f1d865942, store=B 2023-04-15 02:55:22,894 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-15 02:55:22,894 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 104a1f3a98692b3503aef02f1d865942, store=C 2023-04-15 02:55:22,894 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-15 02:55:23,305 INFO [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304154130093084044f1484d00a9ea791bb59 to hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304154130093084044f1484d00a9ea791bb59 2023-04-15 02:55:23,312 INFO [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=320, memsize=17.9 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/104a1f3a98692b3503aef02f1d865942/.tmp/A/eed0c6378f2847a1973423d6b01f9dba 2023-04-15 02:55:23,408 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34681] regionserver.HRegion(9158): Flush requested on 104a1f3a98692b3503aef02f1d865942 2023-04-15 02:55:23,408 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681527301940.104a1f3a98692b3503aef02f1d865942. as already flushing 2023-04-15 02:55:23,503 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681527323464, max timestamp=1681527323501Num uniques 859; 2023-04-15 02:55:23,503 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681527323464, max timestamp=1681527323501Num uniques 859; 2023-04-15 02:55:23,503 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-1] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-15 02:55:23,503 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681527323464, max timestamp=1681527323501Num uniques 859; 2023-04-15 02:55:23,503 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-2] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-15 02:55:23,503 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-7] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-15 02:55:23,579 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=104a1f3a98692b3503aef02f1d865942, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:55:23,579 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34681] ipc.CallRunner(144): callId: 54 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41166 deadline: 1681527383571, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=104a1f3a98692b3503aef02f1d865942, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:23,584 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=104a1f3a98692b3503aef02f1d865942, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:55:23,584 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34681] ipc.CallRunner(144): callId: 291 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:41180 deadline: 1681527383572, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=104a1f3a98692b3503aef02f1d865942, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:23,682 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=104a1f3a98692b3503aef02f1d865942, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:55:23,682 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34681] ipc.CallRunner(144): callId: 56 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41166 deadline: 1681527383680, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=104a1f3a98692b3503aef02f1d865942, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:23,690 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=104a1f3a98692b3503aef02f1d865942, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:55:23,690 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34681] ipc.CallRunner(144): callId: 293 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:41180 deadline: 1681527383685, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=104a1f3a98692b3503aef02f1d865942, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:23,722 INFO [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=320 (bloomFilter=true), to=hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/104a1f3a98692b3503aef02f1d865942/.tmp/B/cac7506f16b2408b80ecf02f1d522752 2023-04-15 02:55:23,889 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=104a1f3a98692b3503aef02f1d865942, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:55:23,889 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34681] ipc.CallRunner(144): callId: 58 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41166 deadline: 1681527383883, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=104a1f3a98692b3503aef02f1d865942, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:23,896 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=104a1f3a98692b3503aef02f1d865942, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:55:23,897 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34681] ipc.CallRunner(144): callId: 295 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:41180 deadline: 1681527383891, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=104a1f3a98692b3503aef02f1d865942, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:24,135 INFO [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=320 (bloomFilter=true), to=hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/104a1f3a98692b3503aef02f1d865942/.tmp/C/69e313b055a240f6828f4ebb73ac2c43 2023-04-15 02:55:24,138 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/104a1f3a98692b3503aef02f1d865942/.tmp/A/eed0c6378f2847a1973423d6b01f9dba as hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/104a1f3a98692b3503aef02f1d865942/A/eed0c6378f2847a1973423d6b01f9dba 2023-04-15 02:55:24,141 INFO [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/104a1f3a98692b3503aef02f1d865942/A/eed0c6378f2847a1973423d6b01f9dba, entries=150, sequenceid=320, filesize=25.4 K 2023-04-15 02:55:24,141 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/104a1f3a98692b3503aef02f1d865942/.tmp/B/cac7506f16b2408b80ecf02f1d522752 as hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/104a1f3a98692b3503aef02f1d865942/B/cac7506f16b2408b80ecf02f1d522752 2023-04-15 02:55:24,144 INFO [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/104a1f3a98692b3503aef02f1d865942/B/cac7506f16b2408b80ecf02f1d522752, entries=150, sequenceid=320, filesize=11.9 K 2023-04-15 02:55:24,144 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/104a1f3a98692b3503aef02f1d865942/.tmp/C/69e313b055a240f6828f4ebb73ac2c43 as hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/104a1f3a98692b3503aef02f1d865942/C/69e313b055a240f6828f4ebb73ac2c43 2023-04-15 02:55:24,147 INFO [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/104a1f3a98692b3503aef02f1d865942/C/69e313b055a240f6828f4ebb73ac2c43, entries=150, sequenceid=320, filesize=11.9 K 2023-04-15 02:55:24,148 INFO [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=127.47 KB/130530 for 104a1f3a98692b3503aef02f1d865942 in 1254ms, sequenceid=320, compaction requested=true 2023-04-15 02:55:24,148 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 104a1f3a98692b3503aef02f1d865942: 2023-04-15 02:55:24,148 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681527301940.104a1f3a98692b3503aef02f1d865942. 2023-04-15 02:55:24,148 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-15 02:55:24,148 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-15 02:55:24,148 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:24,148 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-15 02:55:24,148 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34681,1681527209133' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-15 02:55:24,149 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:24,149 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:24,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:24,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:55:24,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:55:24,150 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:24,150 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-15 02:55:24,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:55:24,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:55:24,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:55:24,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:24,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:55:24,151 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34681,1681527209133' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-15 02:55:24,151 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-15 02:55:24,151 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4c2333a3[Count = 0] remaining members to acquire global barrier 2023-04-15 02:55:24,151 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:24,151 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:24,151 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:24,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:24,151 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-15 02:55:24,152 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-15 02:55:24,152 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34681,1681527209133' in zk 2023-04-15 02:55:24,152 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:24,152 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-15 02:55:24,152 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:24,152 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-15 02:55:24,152 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-15 02:55:24,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:24,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:55:24,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:55:24,152 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-15 02:55:24,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:55:24,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:55:24,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:55:24,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:24,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:55:24,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:55:24,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:24,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34681,1681527209133': 2023-04-15 02:55:24,154 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34681,1681527209133' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-15 02:55:24,154 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-15 02:55:24,154 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-15 02:55:24,154 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-15 02:55:24,154 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-15 02:55:24,154 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-15 02:55:24,155 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:24,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:24,155 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:24,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:55:24,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:55:24,155 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:24,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:24,155 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-15 02:55:24,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:55:24,158 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-15 02:55:24,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-15 02:55:24,158 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:24,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:55:24,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:24,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:55:24,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:55:24,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:24,162 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:24,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:55:24,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:55:24,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:24,171 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:24,171 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:55:24,171 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-15 02:55:24,171 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-15 02:55:24,171 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:55:24,171 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-15 02:55:24,171 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-15 02:55:24,171 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:24,171 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:24,171 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:24,171 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:24,171 DEBUG [Thread-2181] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-15 02:55:24,171 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-15 02:55:24,171 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-15 02:55:24,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-15 02:55:24,171 DEBUG [Thread-2181] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-15 02:55:24,172 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-15 02:55:24,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-15 02:55:24,193 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34681] regionserver.HRegion(9158): Flush requested on 104a1f3a98692b3503aef02f1d865942 2023-04-15 02:55:24,193 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 104a1f3a98692b3503aef02f1d865942 3/3 column families, dataSize=134.18 KB heapSize=384.98 KB 2023-04-15 02:55:24,193 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 104a1f3a98692b3503aef02f1d865942, store=A 2023-04-15 02:55:24,193 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-15 02:55:24,193 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 104a1f3a98692b3503aef02f1d865942, store=B 2023-04-15 02:55:24,193 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-15 02:55:24,193 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 104a1f3a98692b3503aef02f1d865942, store=C 2023-04-15 02:55:24,193 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-15 02:55:24,226 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=104a1f3a98692b3503aef02f1d865942, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:55:24,226 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34681] ipc.CallRunner(144): callId: 300 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:41180 deadline: 1681527384218, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=104a1f3a98692b3503aef02f1d865942, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:24,233 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=104a1f3a98692b3503aef02f1d865942, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:55:24,233 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34681] ipc.CallRunner(144): callId: 65 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41166 deadline: 1681527384224, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=104a1f3a98692b3503aef02f1d865942, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:24,272 DEBUG [Thread-2181] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-15 02:55:24,272 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-15 02:55:24,274 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-15 02:55:24,275 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-15 02:55:24,275 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-15 02:55:24,275 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-15 02:55:24,275 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-15 02:55:24,275 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-15 02:55:24,275 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-15 02:55:24,276 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:24,276 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:55:24,276 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:24,276 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-15 02:55:24,276 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-15 02:55:24,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-15 02:55:24,278 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:24,278 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-15 02:55:24,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:55:24,279 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:24,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-15 02:55:24,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:55:24,280 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-15 02:55:24,280 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-15 02:55:24,280 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-15 02:55:24,280 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-15 02:55:24,280 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-15 02:55:24,280 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-15 02:55:24,281 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-15 02:55:24,281 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-15 02:55:24,281 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681527301940.104a1f3a98692b3503aef02f1d865942. 2023-04-15 02:55:24,281 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681527301940.104a1f3a98692b3503aef02f1d865942. started... 2023-04-15 02:55:24,281 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681527301940.104a1f3a98692b3503aef02f1d865942. as already flushing 2023-04-15 02:55:24,281 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681527301940.104a1f3a98692b3503aef02f1d865942. 2023-04-15 02:55:24,281 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-15 02:55:24,281 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-15 02:55:24,281 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:24,281 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-15 02:55:24,281 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34681,1681527209133' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-15 02:55:24,282 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:24,282 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:24,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:24,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:55:24,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:55:24,282 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:24,282 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-15 02:55:24,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:55:24,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:55:24,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:55:24,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:24,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:55:24,283 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34681,1681527209133' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-15 02:55:24,283 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-15 02:55:24,283 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@464d97c9[Count = 0] remaining members to acquire global barrier 2023-04-15 02:55:24,283 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:24,284 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:24,284 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:24,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:24,284 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:24,284 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-15 02:55:24,284 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-15 02:55:24,284 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-15 02:55:24,284 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34681,1681527209133' in zk 2023-04-15 02:55:24,285 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:24,285 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-15 02:55:24,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:24,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:55:24,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:55:24,285 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-15 02:55:24,285 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-15 02:55:24,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:55:24,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:55:24,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:55:24,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:24,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:55:24,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:55:24,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:24,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34681,1681527209133': 2023-04-15 02:55:24,287 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34681,1681527209133' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-15 02:55:24,287 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-15 02:55:24,287 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-15 02:55:24,287 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-15 02:55:24,287 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-15 02:55:24,287 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-15 02:55:24,288 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:24,288 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-15 02:55:24,288 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:24,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:24,288 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:24,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:24,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:55:24,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:55:24,289 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-15 02:55:24,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-15 02:55:24,289 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:24,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:55:24,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:24,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:55:24,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:55:24,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:55:24,290 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:24,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:24,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:55:24,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:55:24,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:24,292 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:24,292 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:55:24,292 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-15 02:55:24,292 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-15 02:55:24,292 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:55:24,292 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-15 02:55:24,292 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-15 02:55:24,292 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-15 02:55:24,292 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-15 02:55:24,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-15 02:55:24,292 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:24,293 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:24,293 DEBUG [Thread-2181] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-15 02:55:24,293 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:24,293 DEBUG [Thread-2181] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-15 02:55:24,293 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:24,293 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-15 02:55:24,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-15 02:55:24,334 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=104a1f3a98692b3503aef02f1d865942, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:55:24,334 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34681] ipc.CallRunner(144): callId: 302 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:41180 deadline: 1681527384327, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=104a1f3a98692b3503aef02f1d865942, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:24,341 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=104a1f3a98692b3503aef02f1d865942, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:55:24,341 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34681] ipc.CallRunner(144): callId: 67 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41166 deadline: 1681527384334, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=104a1f3a98692b3503aef02f1d865942, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:24,393 DEBUG [Thread-2181] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-15 02:55:24,393 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-15 02:55:24,394 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-15 02:55:24,396 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-15 02:55:24,396 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-15 02:55:24,396 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-15 02:55:24,396 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-15 02:55:24,396 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-15 02:55:24,396 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-15 02:55:24,396 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:24,396 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:55:24,397 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:24,397 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-15 02:55:24,397 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-15 02:55:24,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-15 02:55:24,397 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:24,397 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-15 02:55:24,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:55:24,398 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:24,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-15 02:55:24,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:55:24,398 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-15 02:55:24,398 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-15 02:55:24,398 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-15 02:55:24,398 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-15 02:55:24,398 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-15 02:55:24,398 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-15 02:55:24,398 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-15 02:55:24,398 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681527301940.104a1f3a98692b3503aef02f1d865942. 2023-04-15 02:55:24,399 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681527301940.104a1f3a98692b3503aef02f1d865942. started... 2023-04-15 02:55:24,399 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681527301940.104a1f3a98692b3503aef02f1d865942. as already flushing 2023-04-15 02:55:24,399 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681527301940.104a1f3a98692b3503aef02f1d865942. 2023-04-15 02:55:24,399 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-15 02:55:24,399 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-15 02:55:24,399 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:24,399 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-15 02:55:24,399 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34681,1681527209133' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-15 02:55:24,400 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:24,400 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:24,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:24,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:55:24,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:55:24,400 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:24,400 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-15 02:55:24,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:55:24,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:55:24,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:55:24,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:24,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:55:24,401 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34681,1681527209133' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-15 02:55:24,401 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-15 02:55:24,401 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2592f919[Count = 0] remaining members to acquire global barrier 2023-04-15 02:55:24,401 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:24,402 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:24,402 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:24,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:24,402 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-15 02:55:24,402 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-15 02:55:24,402 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34681,1681527209133' in zk 2023-04-15 02:55:24,403 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:24,403 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-15 02:55:24,403 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-15 02:55:24,403 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-15 02:55:24,403 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-15 02:55:24,403 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:24,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:24,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:55:24,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:55:24,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:55:24,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:55:24,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:55:24,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:24,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:55:24,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:55:24,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:24,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34681,1681527209133': 2023-04-15 02:55:24,406 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34681,1681527209133' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-15 02:55:24,406 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-15 02:55:24,406 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-15 02:55:24,406 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-15 02:55:24,406 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-15 02:55:24,406 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-15 02:55:24,407 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:24,407 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:24,407 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:24,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:24,407 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-15 02:55:24,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:24,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:55:24,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:55:24,407 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-15 02:55:24,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-15 02:55:24,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:55:24,407 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:24,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:24,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:55:24,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:55:24,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:55:24,408 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:24,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:24,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:55:24,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:55:24,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:24,421 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:24,421 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:55:24,421 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:55:24,421 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:24,421 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:24,421 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-15 02:55:24,421 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-15 02:55:24,421 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-15 02:55:24,421 DEBUG [Thread-2181] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-15 02:55:24,421 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-15 02:55:24,422 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-15 02:55:24,421 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:24,422 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-15 02:55:24,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-15 02:55:24,422 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-15 02:55:24,421 DEBUG [Thread-2181] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-15 02:55:24,422 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:24,422 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-15 02:55:24,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-15 02:55:24,522 DEBUG [Thread-2181] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-15 02:55:24,522 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-15 02:55:24,524 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-15 02:55:24,525 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-15 02:55:24,525 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-15 02:55:24,525 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-15 02:55:24,525 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-15 02:55:24,525 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-15 02:55:24,525 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-15 02:55:24,525 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:24,525 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:55:24,526 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-15 02:55:24,526 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:24,526 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-15 02:55:24,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-15 02:55:24,526 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:24,526 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-15 02:55:24,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:55:24,527 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:24,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-15 02:55:24,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:55:24,527 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-15 02:55:24,527 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-15 02:55:24,527 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-15 02:55:24,527 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-15 02:55:24,527 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-15 02:55:24,527 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-15 02:55:24,527 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-15 02:55:24,527 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-15 02:55:24,527 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681527301940.104a1f3a98692b3503aef02f1d865942. 2023-04-15 02:55:24,527 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681527301940.104a1f3a98692b3503aef02f1d865942. started... 2023-04-15 02:55:24,528 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681527301940.104a1f3a98692b3503aef02f1d865942. as already flushing 2023-04-15 02:55:24,528 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681527301940.104a1f3a98692b3503aef02f1d865942. 2023-04-15 02:55:24,528 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-15 02:55:24,528 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-15 02:55:24,528 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:24,528 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-15 02:55:24,528 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34681,1681527209133' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-15 02:55:24,528 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:24,528 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:24,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:24,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:55:24,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:55:24,529 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:24,529 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-15 02:55:24,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:55:24,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:55:24,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:55:24,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:24,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:55:24,530 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34681,1681527209133' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-15 02:55:24,530 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-15 02:55:24,530 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@27937fc0[Count = 0] remaining members to acquire global barrier 2023-04-15 02:55:24,530 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:24,531 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:24,531 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:24,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:24,531 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-15 02:55:24,531 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-15 02:55:24,531 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34681,1681527209133' in zk 2023-04-15 02:55:24,531 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:24,531 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-15 02:55:24,531 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-15 02:55:24,531 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-15 02:55:24,531 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:24,531 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-15 02:55:24,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:24,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:55:24,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:55:24,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:55:24,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:55:24,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:55:24,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:24,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:55:24,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:55:24,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:24,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34681,1681527209133': 2023-04-15 02:55:24,533 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34681,1681527209133' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-15 02:55:24,533 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-15 02:55:24,533 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-15 02:55:24,533 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-15 02:55:24,533 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-15 02:55:24,533 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-15 02:55:24,534 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:24,534 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:24,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:24,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:55:24,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:55:24,534 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-15 02:55:24,534 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:24,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:24,535 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:24,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:55:24,535 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-15 02:55:24,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-15 02:55:24,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:55:24,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:24,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:55:24,536 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:24,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:55:24,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:24,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:55:24,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:55:24,537 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:24,537 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:55:24,537 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-15 02:55:24,537 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-15 02:55:24,537 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-15 02:55:24,537 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:55:24,537 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-15 02:55:24,537 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-15 02:55:24,537 DEBUG [Thread-2181] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-15 02:55:24,537 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:24,537 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-15 02:55:24,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-15 02:55:24,537 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-15 02:55:24,537 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:24,537 DEBUG [Thread-2181] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-15 02:55:24,537 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:24,537 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:24,538 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-15 02:55:24,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-15 02:55:24,540 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=104a1f3a98692b3503aef02f1d865942, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:55:24,540 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34681] ipc.CallRunner(144): callId: 304 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:41180 deadline: 1681527384535, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=104a1f3a98692b3503aef02f1d865942, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:24,549 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=104a1f3a98692b3503aef02f1d865942, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:55:24,549 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34681] ipc.CallRunner(144): callId: 69 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41166 deadline: 1681527384543, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=104a1f3a98692b3503aef02f1d865942, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:24,609 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230415365171c90c824b43b1173a3ec6206609 to hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230415365171c90c824b43b1173a3ec6206609 2023-04-15 02:55:24,638 DEBUG [Thread-2181] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-15 02:55:24,638 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-15 02:55:24,639 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-15 02:55:24,640 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-15 02:55:24,640 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-15 02:55:24,640 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-15 02:55:24,640 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-15 02:55:24,640 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-15 02:55:24,640 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-15 02:55:24,641 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:24,641 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:55:24,641 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:24,641 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-15 02:55:24,641 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-15 02:55:24,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-15 02:55:24,641 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:24,642 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-15 02:55:24,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:55:24,642 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:24,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-15 02:55:24,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:55:24,642 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-15 02:55:24,642 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-15 02:55:24,642 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-15 02:55:24,642 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-15 02:55:24,642 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-15 02:55:24,643 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-15 02:55:24,643 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-15 02:55:24,643 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-15 02:55:24,643 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681527301940.104a1f3a98692b3503aef02f1d865942. 2023-04-15 02:55:24,643 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681527301940.104a1f3a98692b3503aef02f1d865942. started... 2023-04-15 02:55:24,643 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681527301940.104a1f3a98692b3503aef02f1d865942. as already flushing 2023-04-15 02:55:24,643 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681527301940.104a1f3a98692b3503aef02f1d865942. 2023-04-15 02:55:24,643 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-15 02:55:24,643 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-15 02:55:24,643 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:24,643 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-15 02:55:24,643 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34681,1681527209133' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-15 02:55:24,644 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:24,644 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:24,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:24,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:55:24,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:55:24,644 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:24,644 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-15 02:55:24,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:55:24,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:55:24,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:55:24,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:24,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:55:24,645 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34681,1681527209133' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-15 02:55:24,645 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-15 02:55:24,645 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@72f8eeba[Count = 0] remaining members to acquire global barrier 2023-04-15 02:55:24,645 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:24,646 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:24,646 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:24,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:24,646 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-15 02:55:24,646 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-15 02:55:24,646 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34681,1681527209133' in zk 2023-04-15 02:55:24,646 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:24,646 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-15 02:55:24,647 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-15 02:55:24,647 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-15 02:55:24,647 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:24,647 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-15 02:55:24,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:24,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:55:24,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:55:24,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:55:24,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:55:24,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:55:24,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:24,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:55:24,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:55:24,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:24,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34681,1681527209133': 2023-04-15 02:55:24,649 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34681,1681527209133' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-15 02:55:24,649 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-15 02:55:24,649 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-15 02:55:24,649 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-15 02:55:24,649 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-15 02:55:24,649 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-15 02:55:24,650 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:24,650 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:24,650 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-15 02:55:24,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:24,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:55:24,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:55:24,650 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:24,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:24,650 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:24,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:55:24,650 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-15 02:55:24,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-15 02:55:24,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:55:24,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:24,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:55:24,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:55:24,651 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:24,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:24,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:55:24,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:55:24,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:24,653 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-15 02:55:24,653 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:24,653 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-15 02:55:24,653 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-15 02:55:24,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-15 02:55:24,653 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-15 02:55:24,654 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-15 02:55:24,653 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-15 02:55:24,654 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-15 02:55:24,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-15 02:55:24,653 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:55:24,654 DEBUG [Thread-2181] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-15 02:55:24,654 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:55:24,654 DEBUG [Thread-2181] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-15 02:55:24,654 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:24,654 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:24,654 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:24,654 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:24,754 DEBUG [Thread-2181] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-15 02:55:24,755 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-15 02:55:24,756 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-15 02:55:24,757 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-15 02:55:24,757 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-15 02:55:24,757 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-15 02:55:24,757 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-15 02:55:24,757 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-15 02:55:24,757 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-15 02:55:24,757 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:24,757 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:55:24,758 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-15 02:55:24,758 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:24,758 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-15 02:55:24,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-15 02:55:24,758 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:24,758 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-15 02:55:24,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:55:24,759 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:24,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-15 02:55:24,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:55:24,759 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-15 02:55:24,759 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-15 02:55:24,759 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-15 02:55:24,759 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-15 02:55:24,759 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-15 02:55:24,759 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-15 02:55:24,759 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-15 02:55:24,759 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-15 02:55:24,759 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681527301940.104a1f3a98692b3503aef02f1d865942. 2023-04-15 02:55:24,759 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681527301940.104a1f3a98692b3503aef02f1d865942. started... 2023-04-15 02:55:24,759 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681527301940.104a1f3a98692b3503aef02f1d865942. as already flushing 2023-04-15 02:55:24,759 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681527301940.104a1f3a98692b3503aef02f1d865942. 2023-04-15 02:55:24,759 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-15 02:55:24,759 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-15 02:55:24,760 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:24,760 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-15 02:55:24,760 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34681,1681527209133' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-15 02:55:24,760 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:24,760 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:24,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:24,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:55:24,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:55:24,761 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:24,761 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-15 02:55:24,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:55:24,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:55:24,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:55:24,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:24,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:55:24,761 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34681,1681527209133' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-15 02:55:24,762 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-15 02:55:24,762 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5821dcf3[Count = 0] remaining members to acquire global barrier 2023-04-15 02:55:24,762 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:24,762 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:24,762 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:24,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:24,762 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-15 02:55:24,762 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-15 02:55:24,762 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:24,762 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-15 02:55:24,762 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34681,1681527209133' in zk 2023-04-15 02:55:24,763 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:24,763 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-15 02:55:24,763 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-15 02:55:24,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:24,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:55:24,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:55:24,763 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-15 02:55:24,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:55:24,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:55:24,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:55:24,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:24,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:55:24,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:55:24,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:24,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34681,1681527209133': 2023-04-15 02:55:24,765 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34681,1681527209133' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-15 02:55:24,765 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-15 02:55:24,765 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-15 02:55:24,765 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-15 02:55:24,765 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-15 02:55:24,765 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-15 02:55:24,766 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:24,766 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:24,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:24,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:55:24,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:55:24,766 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:24,766 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-15 02:55:24,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:24,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:55:24,766 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-15 02:55:24,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:55:24,766 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:24,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-15 02:55:24,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:55:24,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:24,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:55:24,767 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:24,768 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:24,768 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-15 02:55:24,768 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:55:24,768 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:55:24,768 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-15 02:55:24,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-15 02:55:24,768 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-15 02:55:24,768 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:24,768 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:24,768 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:24,768 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:24,768 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-15 02:55:24,768 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-15 02:55:24,768 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-15 02:55:24,769 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-15 02:55:24,769 DEBUG [Thread-2181] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-15 02:55:24,769 DEBUG [Thread-2181] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-15 02:55:24,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-15 02:55:24,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:24,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:55:24,770 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-15 02:55:24,848 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=104a1f3a98692b3503aef02f1d865942, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:55:24,848 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34681] ipc.CallRunner(144): callId: 306 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:41180 deadline: 1681527384842, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=104a1f3a98692b3503aef02f1d865942, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:24,859 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=104a1f3a98692b3503aef02f1d865942, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:55:24,859 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34681] ipc.CallRunner(144): callId: 71 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41166 deadline: 1681527384850, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=104a1f3a98692b3503aef02f1d865942, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:24,869 DEBUG [Thread-2181] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-15 02:55:24,869 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-15 02:55:24,870 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-15 02:55:24,871 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-15 02:55:24,871 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-15 02:55:24,871 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-15 02:55:24,871 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-15 02:55:24,871 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-15 02:55:24,871 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-15 02:55:24,871 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:24,872 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:55:24,872 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-15 02:55:24,872 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:24,872 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-15 02:55:24,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-15 02:55:24,872 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:24,872 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-15 02:55:24,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:55:24,873 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:24,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-15 02:55:24,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:55:24,873 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-15 02:55:24,873 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-15 02:55:24,873 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-15 02:55:24,873 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-15 02:55:24,873 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-15 02:55:24,873 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-15 02:55:24,873 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-15 02:55:24,873 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-15 02:55:24,873 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681527301940.104a1f3a98692b3503aef02f1d865942. 2023-04-15 02:55:24,873 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681527301940.104a1f3a98692b3503aef02f1d865942. started... 2023-04-15 02:55:24,874 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681527301940.104a1f3a98692b3503aef02f1d865942. as already flushing 2023-04-15 02:55:24,874 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681527301940.104a1f3a98692b3503aef02f1d865942. 2023-04-15 02:55:24,874 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-15 02:55:24,874 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-15 02:55:24,874 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:24,874 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-15 02:55:24,874 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34681,1681527209133' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-15 02:55:24,875 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:24,875 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:24,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:24,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:55:24,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:55:24,875 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:24,875 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-15 02:55:24,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:55:24,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:55:24,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:55:24,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:24,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:55:24,876 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34681,1681527209133' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-15 02:55:24,876 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-15 02:55:24,876 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6ac9acdf[Count = 0] remaining members to acquire global barrier 2023-04-15 02:55:24,876 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:24,877 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:24,877 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:24,877 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:24,877 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-15 02:55:24,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:24,877 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-15 02:55:24,877 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-15 02:55:24,877 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34681,1681527209133' in zk 2023-04-15 02:55:24,878 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:24,878 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-15 02:55:24,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:24,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:55:24,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:55:24,878 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-15 02:55:24,878 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-15 02:55:24,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:55:24,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:55:24,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:55:24,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:24,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:55:24,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:55:24,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:24,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34681,1681527209133': 2023-04-15 02:55:24,880 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34681,1681527209133' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-15 02:55:24,880 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-15 02:55:24,880 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-15 02:55:24,880 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-15 02:55:24,880 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-15 02:55:24,880 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-15 02:55:24,881 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:24,881 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:24,881 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-15 02:55:24,881 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:24,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:24,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:55:24,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:55:24,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:24,881 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:24,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:55:24,881 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-15 02:55:24,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-15 02:55:24,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:55:24,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:24,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:55:24,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:55:24,882 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:24,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:24,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:55:24,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:55:24,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:24,883 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-15 02:55:24,883 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-15 02:55:24,883 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-15 02:55:24,883 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-15 02:55:24,883 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-15 02:55:24,883 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-15 02:55:24,884 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-15 02:55:24,883 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:24,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-15 02:55:24,884 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:55:24,884 DEBUG [Thread-2181] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-15 02:55:24,884 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:55:24,884 DEBUG [Thread-2181] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-15 02:55:24,884 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:24,884 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-15 02:55:24,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-15 02:55:24,884 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:24,884 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:24,884 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:24,984 DEBUG [Thread-2181] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-15 02:55:24,984 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-15 02:55:24,986 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-15 02:55:24,986 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-15 02:55:24,986 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-15 02:55:24,987 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-15 02:55:24,987 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-15 02:55:24,987 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-15 02:55:24,987 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-15 02:55:24,987 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:24,987 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:55:24,988 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:24,988 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-15 02:55:24,988 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-15 02:55:24,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-15 02:55:24,988 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:24,988 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-15 02:55:24,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:55:24,988 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:24,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-15 02:55:24,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:55:24,989 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-15 02:55:24,989 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-15 02:55:24,989 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-15 02:55:24,989 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-15 02:55:24,989 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-15 02:55:24,989 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-15 02:55:24,989 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-15 02:55:24,989 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-15 02:55:24,989 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681527301940.104a1f3a98692b3503aef02f1d865942. 2023-04-15 02:55:24,989 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681527301940.104a1f3a98692b3503aef02f1d865942. started... 2023-04-15 02:55:24,989 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681527301940.104a1f3a98692b3503aef02f1d865942. as already flushing 2023-04-15 02:55:24,989 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681527301940.104a1f3a98692b3503aef02f1d865942. 2023-04-15 02:55:24,989 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-15 02:55:24,989 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-15 02:55:24,989 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:24,989 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-15 02:55:24,989 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34681,1681527209133' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-15 02:55:24,990 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:24,990 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:24,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:24,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:55:24,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:55:24,990 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:24,990 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-15 02:55:24,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:55:24,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:55:24,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:55:24,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:24,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:55:24,991 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34681,1681527209133' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-15 02:55:24,991 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5618b339[Count = 0] remaining members to acquire global barrier 2023-04-15 02:55:24,991 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-15 02:55:24,991 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:24,992 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:24,992 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:24,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:24,992 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-15 02:55:24,992 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-15 02:55:24,992 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34681,1681527209133' in zk 2023-04-15 02:55:24,992 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:24,992 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-15 02:55:24,993 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:24,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:24,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:55:24,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:55:24,993 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-15 02:55:24,993 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-15 02:55:24,993 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-15 02:55:24,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:55:24,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:55:24,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:55:24,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:24,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:55:24,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:55:24,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:24,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34681,1681527209133': 2023-04-15 02:55:24,995 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-15 02:55:24,995 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-15 02:55:24,995 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-15 02:55:24,995 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34681,1681527209133' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-15 02:55:24,995 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-15 02:55:24,995 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-15 02:55:24,996 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:24,996 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:24,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:24,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:55:24,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:55:24,996 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-15 02:55:24,996 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:24,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:24,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:55:24,996 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:24,996 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-15 02:55:24,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-15 02:55:24,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:24,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:55:24,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:55:24,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:55:24,997 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:24,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:24,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:55:24,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:55:24,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:24,998 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:24,998 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-15 02:55:24,998 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:55:24,998 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:55:24,998 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:24,998 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:24,998 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-15 02:55:24,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-15 02:55:24,998 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-15 02:55:24,998 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:24,999 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:24,998 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-15 02:55:24,999 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-15 02:55:24,998 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-15 02:55:24,999 DEBUG [Thread-2181] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-15 02:55:24,999 DEBUG [Thread-2181] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-15 02:55:24,999 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-15 02:55:24,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-15 02:55:25,000 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-15 02:55:25,018 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=357, memsize=44.7 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/104a1f3a98692b3503aef02f1d865942/.tmp/A/e199d5e34d4a4eefb87d3a288bdfd58b 2023-04-15 02:55:25,099 DEBUG [Thread-2190] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x5ba3ce53 to 127.0.0.1:51762 2023-04-15 02:55:25,099 DEBUG [Thread-2190] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-15 02:55:25,099 DEBUG [Thread-2186] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x060c5cba to 127.0.0.1:51762 2023-04-15 02:55:25,099 DEBUG [Thread-2186] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-15 02:55:25,099 DEBUG [Thread-2181] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-15 02:55:25,100 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-15 02:55:25,105 DEBUG [Thread-2182] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x5da5030e to 127.0.0.1:51762 2023-04-15 02:55:25,105 DEBUG [Thread-2182] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-15 02:55:25,110 DEBUG [Thread-2188] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x52249391 to 127.0.0.1:51762 2023-04-15 02:55:25,110 DEBUG [Thread-2188] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-15 02:55:25,110 DEBUG [Thread-2184] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x74d6bf18 to 127.0.0.1:51762 2023-04-15 02:55:25,110 DEBUG [Thread-2184] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-15 02:55:25,349 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=104a1f3a98692b3503aef02f1d865942, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:55:25,349 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34681] ipc.CallRunner(144): callId: 308 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:41180 deadline: 1681527385349, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=104a1f3a98692b3503aef02f1d865942, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:25,364 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=104a1f3a98692b3503aef02f1d865942, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:55:25,364 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34681] ipc.CallRunner(144): callId: 73 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41166 deadline: 1681527385364, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=104a1f3a98692b3503aef02f1d865942, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:25,428 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=357 (bloomFilter=true), to=hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/104a1f3a98692b3503aef02f1d865942/.tmp/B/dea8ed53bbc34af9bcd3d8a21ada58c4 2023-04-15 02:55:25,845 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=357 (bloomFilter=true), to=hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/104a1f3a98692b3503aef02f1d865942/.tmp/C/007d8f3a743146b18a51a5a03aa59d2c 2023-04-15 02:55:25,853 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/104a1f3a98692b3503aef02f1d865942/.tmp/A/e199d5e34d4a4eefb87d3a288bdfd58b as hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/104a1f3a98692b3503aef02f1d865942/A/e199d5e34d4a4eefb87d3a288bdfd58b 2023-04-15 02:55:25,860 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/104a1f3a98692b3503aef02f1d865942/A/e199d5e34d4a4eefb87d3a288bdfd58b, entries=200, sequenceid=357, filesize=32.3 K 2023-04-15 02:55:25,862 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/104a1f3a98692b3503aef02f1d865942/.tmp/B/dea8ed53bbc34af9bcd3d8a21ada58c4 as hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/104a1f3a98692b3503aef02f1d865942/B/dea8ed53bbc34af9bcd3d8a21ada58c4 2023-04-15 02:55:25,869 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/104a1f3a98692b3503aef02f1d865942/B/dea8ed53bbc34af9bcd3d8a21ada58c4, entries=150, sequenceid=357, filesize=11.9 K 2023-04-15 02:55:25,870 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/104a1f3a98692b3503aef02f1d865942/.tmp/C/007d8f3a743146b18a51a5a03aa59d2c as hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/104a1f3a98692b3503aef02f1d865942/C/007d8f3a743146b18a51a5a03aa59d2c 2023-04-15 02:55:25,877 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/104a1f3a98692b3503aef02f1d865942/C/007d8f3a743146b18a51a5a03aa59d2c, entries=150, sequenceid=357, filesize=11.9 K 2023-04-15 02:55:25,877 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~134.18 KB/137400, heapSize ~384.94 KB/394176, currentSize=46.96 KB/48090 for 104a1f3a98692b3503aef02f1d865942 in 1684ms, sequenceid=357, compaction requested=true 2023-04-15 02:55:25,878 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 104a1f3a98692b3503aef02f1d865942: 2023-04-15 02:55:25,878 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-15 02:55:25,878 DEBUG [RS:0;jenkins-hbase20:34681-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 7 store files, 0 compacting, 7 eligible, 16 blocking 2023-04-15 02:55:25,878 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-15 02:55:25,878 DEBUG [RS:0;jenkins-hbase20:34681-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 7 store files, 0 compacting, 7 eligible, 16 blocking 2023-04-15 02:55:25,878 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-15 02:55:25,880 DEBUG [RS:0;jenkins-hbase20:34681-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 7 files of size 196297 starting at candidate #0 after considering 15 permutations with 15 in ratio 2023-04-15 02:55:25,880 DEBUG [RS:0;jenkins-hbase20:34681-shortCompactions-0] regionserver.HStore(1912): 104a1f3a98692b3503aef02f1d865942/A is initiating minor compaction (all files) 2023-04-15 02:55:25,880 DEBUG [RS:0;jenkins-hbase20:34681-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 7 files of size 85247 starting at candidate #0 after considering 15 permutations with 15 in ratio 2023-04-15 02:55:25,880 INFO [RS:0;jenkins-hbase20:34681-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 104a1f3a98692b3503aef02f1d865942/A in TestAcidGuarantees,,1681527301940.104a1f3a98692b3503aef02f1d865942. 2023-04-15 02:55:25,880 DEBUG [RS:0;jenkins-hbase20:34681-longCompactions-0] regionserver.HStore(1912): 104a1f3a98692b3503aef02f1d865942/B is initiating minor compaction (all files) 2023-04-15 02:55:25,880 INFO [RS:0;jenkins-hbase20:34681-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 104a1f3a98692b3503aef02f1d865942/B in TestAcidGuarantees,,1681527301940.104a1f3a98692b3503aef02f1d865942. 2023-04-15 02:55:25,880 INFO [RS:0;jenkins-hbase20:34681-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/104a1f3a98692b3503aef02f1d865942/A/2baf48876e504d309b0eb53ae2597104, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/104a1f3a98692b3503aef02f1d865942/A/33eb33ab038945ffaaa7258b29a920e3, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/104a1f3a98692b3503aef02f1d865942/A/b9c13adc25ec4fe5b85c9fd9f11b5c24, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/104a1f3a98692b3503aef02f1d865942/A/5b535cd666c047c490051e7d46a2e5b2, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/104a1f3a98692b3503aef02f1d865942/A/f891948ce31f453d8d3475787916ed1c, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/104a1f3a98692b3503aef02f1d865942/A/eed0c6378f2847a1973423d6b01f9dba, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/104a1f3a98692b3503aef02f1d865942/A/e199d5e34d4a4eefb87d3a288bdfd58b] into tmpdir=hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/104a1f3a98692b3503aef02f1d865942/.tmp, totalSize=191.7 K 2023-04-15 02:55:25,880 INFO [RS:0;jenkins-hbase20:34681-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/104a1f3a98692b3503aef02f1d865942/B/a62cf6cb551c4811ad937d6d3921e413, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/104a1f3a98692b3503aef02f1d865942/B/8b351a9d1f4740429641269e5aaf1ba4, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/104a1f3a98692b3503aef02f1d865942/B/3551e48afafc4d49a049bfee2388e032, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/104a1f3a98692b3503aef02f1d865942/B/545eb950cd234d8db5b32be8dfbb9a99, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/104a1f3a98692b3503aef02f1d865942/B/343a248dec4643f4b11379095dd65595, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/104a1f3a98692b3503aef02f1d865942/B/cac7506f16b2408b80ecf02f1d522752, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/104a1f3a98692b3503aef02f1d865942/B/dea8ed53bbc34af9bcd3d8a21ada58c4] into tmpdir=hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/104a1f3a98692b3503aef02f1d865942/.tmp, totalSize=83.2 K 2023-04-15 02:55:25,880 DEBUG [RS:0;jenkins-hbase20:34681-shortCompactions-0] compactions.Compactor(207): Compacting 2baf48876e504d309b0eb53ae2597104, keycount=150, bloomtype=ROW, size=25.2 K, encoding=NONE, compression=NONE, seqNum=62, earliestPutTs=1681527306309 2023-04-15 02:55:25,881 DEBUG [RS:0;jenkins-hbase20:34681-longCompactions-0] compactions.Compactor(207): Compacting a62cf6cb551c4811ad937d6d3921e413, keycount=150, bloomtype=ROW, size=11.7 K, encoding=NONE, compression=NONE, seqNum=62, earliestPutTs=1681527306309 2023-04-15 02:55:25,881 DEBUG [RS:0;jenkins-hbase20:34681-shortCompactions-0] compactions.Compactor(207): Compacting 33eb33ab038945ffaaa7258b29a920e3, keycount=150, bloomtype=ROW, size=25.6 K, encoding=NONE, compression=NONE, seqNum=238, earliestPutTs=1681527316192 2023-04-15 02:55:25,881 DEBUG [RS:0;jenkins-hbase20:34681-longCompactions-0] compactions.Compactor(207): Compacting 8b351a9d1f4740429641269e5aaf1ba4, keycount=150, bloomtype=ROW, size=12.1 K, encoding=NONE, compression=NONE, seqNum=238, earliestPutTs=1681527316192 2023-04-15 02:55:25,881 DEBUG [RS:0;jenkins-hbase20:34681-shortCompactions-0] compactions.Compactor(207): Compacting b9c13adc25ec4fe5b85c9fd9f11b5c24, keycount=200, bloomtype=ROW, size=32.2 K, encoding=NONE, compression=NONE, seqNum=259, earliestPutTs=1681527318795 2023-04-15 02:55:25,881 DEBUG [RS:0;jenkins-hbase20:34681-longCompactions-0] compactions.Compactor(207): Compacting 3551e48afafc4d49a049bfee2388e032, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=259, earliestPutTs=1681527318818 2023-04-15 02:55:25,881 DEBUG [RS:0;jenkins-hbase20:34681-shortCompactions-0] compactions.Compactor(207): Compacting 5b535cd666c047c490051e7d46a2e5b2, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=270, earliestPutTs=1681527319159 2023-04-15 02:55:25,881 DEBUG [RS:0;jenkins-hbase20:34681-longCompactions-0] compactions.Compactor(207): Compacting 545eb950cd234d8db5b32be8dfbb9a99, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=270, earliestPutTs=1681527319159 2023-04-15 02:55:25,882 DEBUG [RS:0;jenkins-hbase20:34681-shortCompactions-0] compactions.Compactor(207): Compacting f891948ce31f453d8d3475787916ed1c, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=309, earliestPutTs=1681527320146 2023-04-15 02:55:25,882 DEBUG [RS:0;jenkins-hbase20:34681-longCompactions-0] compactions.Compactor(207): Compacting 343a248dec4643f4b11379095dd65595, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=309, earliestPutTs=1681527320146 2023-04-15 02:55:25,882 DEBUG [RS:0;jenkins-hbase20:34681-shortCompactions-0] compactions.Compactor(207): Compacting eed0c6378f2847a1973423d6b01f9dba, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=320, earliestPutTs=1681527321351 2023-04-15 02:55:25,882 DEBUG [RS:0;jenkins-hbase20:34681-longCompactions-0] compactions.Compactor(207): Compacting cac7506f16b2408b80ecf02f1d522752, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=320, earliestPutTs=1681527321351 2023-04-15 02:55:25,882 DEBUG [RS:0;jenkins-hbase20:34681-shortCompactions-0] compactions.Compactor(207): Compacting e199d5e34d4a4eefb87d3a288bdfd58b, keycount=200, bloomtype=ROW, size=32.3 K, encoding=NONE, compression=NONE, seqNum=357, earliestPutTs=1681527323541 2023-04-15 02:55:25,883 DEBUG [RS:0;jenkins-hbase20:34681-longCompactions-0] compactions.Compactor(207): Compacting dea8ed53bbc34af9bcd3d8a21ada58c4, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=357, earliestPutTs=1681527323541 2023-04-15 02:55:25,895 INFO [RS:0;jenkins-hbase20:34681-longCompactions-0] throttle.PressureAwareThroughputController(145): 104a1f3a98692b3503aef02f1d865942#B#compaction#336 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-15 02:55:25,898 INFO [RS:0;jenkins-hbase20:34681-shortCompactions-0] throttle.PressureAwareThroughputController(145): 104a1f3a98692b3503aef02f1d865942#A#compaction#335 average throughput is 4.93 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-15 02:55:26,311 DEBUG [RS:0;jenkins-hbase20:34681-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/104a1f3a98692b3503aef02f1d865942/.tmp/B/1f9d45a03b974d8d9ca9d12beffd35e7 as hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/104a1f3a98692b3503aef02f1d865942/B/1f9d45a03b974d8d9ca9d12beffd35e7 2023-04-15 02:55:26,316 INFO [RS:0;jenkins-hbase20:34681-longCompactions-0] regionserver.HStore(1652): Completed compaction of 7 (all) file(s) in 104a1f3a98692b3503aef02f1d865942/B of 104a1f3a98692b3503aef02f1d865942 into 1f9d45a03b974d8d9ca9d12beffd35e7(size=12.6 K), total size for store is 12.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-15 02:55:26,316 DEBUG [RS:0;jenkins-hbase20:34681-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 104a1f3a98692b3503aef02f1d865942: 2023-04-15 02:55:26,316 INFO [RS:0;jenkins-hbase20:34681-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681527301940.104a1f3a98692b3503aef02f1d865942., storeName=104a1f3a98692b3503aef02f1d865942/B, priority=9, startTime=1681527325878; duration=0sec 2023-04-15 02:55:26,316 DEBUG [RS:0;jenkins-hbase20:34681-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-15 02:55:26,316 DEBUG [RS:0;jenkins-hbase20:34681-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 7 store files, 0 compacting, 7 eligible, 16 blocking 2023-04-15 02:55:26,319 DEBUG [RS:0;jenkins-hbase20:34681-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 7 files of size 85247 starting at candidate #0 after considering 15 permutations with 15 in ratio 2023-04-15 02:55:26,319 DEBUG [RS:0;jenkins-hbase20:34681-longCompactions-0] regionserver.HStore(1912): 104a1f3a98692b3503aef02f1d865942/C is initiating minor compaction (all files) 2023-04-15 02:55:26,319 INFO [RS:0;jenkins-hbase20:34681-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 104a1f3a98692b3503aef02f1d865942/C in TestAcidGuarantees,,1681527301940.104a1f3a98692b3503aef02f1d865942. 2023-04-15 02:55:26,319 INFO [RS:0;jenkins-hbase20:34681-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/104a1f3a98692b3503aef02f1d865942/C/355d44d68c824843ad7c1f9e8f03cfe4, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/104a1f3a98692b3503aef02f1d865942/C/486f243a5a8a47bfb55d6c75e878658d, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/104a1f3a98692b3503aef02f1d865942/C/41866252f56145e09d7441da4e316361, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/104a1f3a98692b3503aef02f1d865942/C/0ce75ec5e15543de8052b6a6b4e68ce1, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/104a1f3a98692b3503aef02f1d865942/C/2005670e61c1453484cab603e00c404c, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/104a1f3a98692b3503aef02f1d865942/C/69e313b055a240f6828f4ebb73ac2c43, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/104a1f3a98692b3503aef02f1d865942/C/007d8f3a743146b18a51a5a03aa59d2c] into tmpdir=hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/104a1f3a98692b3503aef02f1d865942/.tmp, totalSize=83.2 K 2023-04-15 02:55:26,319 DEBUG [RS:0;jenkins-hbase20:34681-longCompactions-0] compactions.Compactor(207): Compacting 355d44d68c824843ad7c1f9e8f03cfe4, keycount=150, bloomtype=ROW, size=11.7 K, encoding=NONE, compression=NONE, seqNum=62, earliestPutTs=1681527306309 2023-04-15 02:55:26,320 DEBUG [RS:0;jenkins-hbase20:34681-longCompactions-0] compactions.Compactor(207): Compacting 486f243a5a8a47bfb55d6c75e878658d, keycount=150, bloomtype=ROW, size=12.1 K, encoding=NONE, compression=NONE, seqNum=238, earliestPutTs=1681527316192 2023-04-15 02:55:26,320 DEBUG [RS:0;jenkins-hbase20:34681-longCompactions-0] compactions.Compactor(207): Compacting 41866252f56145e09d7441da4e316361, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=259, earliestPutTs=1681527318818 2023-04-15 02:55:26,320 DEBUG [RS:0;jenkins-hbase20:34681-longCompactions-0] compactions.Compactor(207): Compacting 0ce75ec5e15543de8052b6a6b4e68ce1, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=270, earliestPutTs=1681527319159 2023-04-15 02:55:26,321 DEBUG [RS:0;jenkins-hbase20:34681-longCompactions-0] compactions.Compactor(207): Compacting 2005670e61c1453484cab603e00c404c, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=309, earliestPutTs=1681527320146 2023-04-15 02:55:26,321 DEBUG [RS:0;jenkins-hbase20:34681-longCompactions-0] compactions.Compactor(207): Compacting 69e313b055a240f6828f4ebb73ac2c43, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=320, earliestPutTs=1681527321351 2023-04-15 02:55:26,321 DEBUG [RS:0;jenkins-hbase20:34681-longCompactions-0] compactions.Compactor(207): Compacting 007d8f3a743146b18a51a5a03aa59d2c, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=357, earliestPutTs=1681527323541 2023-04-15 02:55:26,335 INFO [RS:0;jenkins-hbase20:34681-longCompactions-0] throttle.PressureAwareThroughputController(145): 104a1f3a98692b3503aef02f1d865942#C#compaction#337 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-15 02:55:26,358 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34681] regionserver.HRegion(9158): Flush requested on 104a1f3a98692b3503aef02f1d865942 2023-04-15 02:55:26,358 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 104a1f3a98692b3503aef02f1d865942 3/3 column families, dataSize=53.67 KB heapSize=160.13 KB 2023-04-15 02:55:26,359 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 104a1f3a98692b3503aef02f1d865942, store=A 2023-04-15 02:55:26,359 DEBUG [Thread-2175] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x377ba302 to 127.0.0.1:51762 2023-04-15 02:55:26,359 DEBUG [Thread-2175] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-15 02:55:26,359 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-15 02:55:26,359 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 104a1f3a98692b3503aef02f1d865942, store=B 2023-04-15 02:55:26,359 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-15 02:55:26,360 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 104a1f3a98692b3503aef02f1d865942, store=C 2023-04-15 02:55:26,360 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-15 02:55:26,383 DEBUG [Thread-2179] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x1d63a8ce to 127.0.0.1:51762 2023-04-15 02:55:26,383 DEBUG [Thread-2179] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-15 02:55:26,747 DEBUG [RS:0;jenkins-hbase20:34681-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/104a1f3a98692b3503aef02f1d865942/.tmp/C/d79e25fb2453479d810a799f1c72d6ff as hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/104a1f3a98692b3503aef02f1d865942/C/d79e25fb2453479d810a799f1c72d6ff 2023-04-15 02:55:26,752 INFO [RS:0;jenkins-hbase20:34681-longCompactions-0] regionserver.HStore(1652): Completed compaction of 7 (all) file(s) in 104a1f3a98692b3503aef02f1d865942/C of 104a1f3a98692b3503aef02f1d865942 into d79e25fb2453479d810a799f1c72d6ff(size=12.6 K), total size for store is 12.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-15 02:55:26,752 DEBUG [RS:0;jenkins-hbase20:34681-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 104a1f3a98692b3503aef02f1d865942: 2023-04-15 02:55:26,752 INFO [RS:0;jenkins-hbase20:34681-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681527301940.104a1f3a98692b3503aef02f1d865942., storeName=104a1f3a98692b3503aef02f1d865942/C, priority=9, startTime=1681527325878; duration=0sec 2023-04-15 02:55:26,752 DEBUG [RS:0;jenkins-hbase20:34681-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-15 02:55:26,779 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304156f2e6c4142a449d785b4994773d2fcca to hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304156f2e6c4142a449d785b4994773d2fcca 2023-04-15 02:55:26,846 DEBUG [Thread-2171] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x6fa3f5c5 to 127.0.0.1:51762 2023-04-15 02:55:26,846 DEBUG [Thread-2171] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-15 02:55:26,880 DEBUG [Thread-2177] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x1eaa86ba to 127.0.0.1:51762 2023-04-15 02:55:26,880 DEBUG [Thread-2177] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-15 02:55:26,897 DEBUG [Thread-2173] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x6ba43635 to 127.0.0.1:51762 2023-04-15 02:55:26,897 DEBUG [Thread-2173] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-15 02:55:26,898 INFO [Listener at localhost.localdomain/33693] hbase.AcidGuaranteesTestTool(392): Finished test. Writers: 2023-04-15 02:55:26,898 INFO [Listener at localhost.localdomain/33693] hbase.AcidGuaranteesTestTool(394): wrote 30 2023-04-15 02:55:26,898 INFO [Listener at localhost.localdomain/33693] hbase.AcidGuaranteesTestTool(394): wrote 24 2023-04-15 02:55:26,898 INFO [Listener at localhost.localdomain/33693] hbase.AcidGuaranteesTestTool(394): wrote 183 2023-04-15 02:55:26,898 INFO [Listener at localhost.localdomain/33693] hbase.AcidGuaranteesTestTool(394): wrote 26 2023-04-15 02:55:26,898 INFO [Listener at localhost.localdomain/33693] hbase.AcidGuaranteesTestTool(394): wrote 42 2023-04-15 02:55:26,899 INFO [Listener at localhost.localdomain/33693] hbase.AcidGuaranteesTestTool(396): Readers: 2023-04-15 02:55:26,899 INFO [Listener at localhost.localdomain/33693] hbase.AcidGuaranteesTestTool(400): Scanners: 2023-04-15 02:55:26,899 INFO [Listener at localhost.localdomain/33693] hbase.AcidGuaranteesTestTool(402): scanned 1530 2023-04-15 02:55:26,899 INFO [Listener at localhost.localdomain/33693] hbase.AcidGuaranteesTestTool(403): verified 4586 rows 2023-04-15 02:55:26,899 INFO [Listener at localhost.localdomain/33693] hbase.AcidGuaranteesTestTool(402): scanned 1523 2023-04-15 02:55:26,899 INFO [Listener at localhost.localdomain/33693] hbase.AcidGuaranteesTestTool(403): verified 4566 rows 2023-04-15 02:55:26,899 INFO [Listener at localhost.localdomain/33693] hbase.AcidGuaranteesTestTool(402): scanned 1530 2023-04-15 02:55:26,899 INFO [Listener at localhost.localdomain/33693] hbase.AcidGuaranteesTestTool(403): verified 4590 rows 2023-04-15 02:55:26,900 INFO [Listener at localhost.localdomain/33693] hbase.AcidGuaranteesTestTool(402): scanned 1526 2023-04-15 02:55:26,900 INFO [Listener at localhost.localdomain/33693] hbase.AcidGuaranteesTestTool(403): verified 4577 rows 2023-04-15 02:55:26,900 INFO [Listener at localhost.localdomain/33693] hbase.AcidGuaranteesTestTool(402): scanned 1525 2023-04-15 02:55:26,900 INFO [Listener at localhost.localdomain/33693] hbase.AcidGuaranteesTestTool(403): verified 4575 rows 2023-04-15 02:55:26,900 INFO [Listener at localhost.localdomain/33693] client.ConnectionImplementation(1974): Closing master protocol: MasterService 2023-04-15 02:55:26,900 DEBUG [Listener at localhost.localdomain/33693] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x7a0240a4 to 127.0.0.1:51762 2023-04-15 02:55:26,900 DEBUG [Listener at localhost.localdomain/33693] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-15 02:55:26,902 INFO [Listener at localhost.localdomain/33693] client.HBaseAdmin$15(890): Started disable of TestAcidGuarantees 2023-04-15 02:55:26,903 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.HMaster$11(2418): Client=jenkins//148.251.75.209 disable TestAcidGuarantees 2023-04-15 02:55:26,904 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] procedure2.ProcedureExecutor(1029): Stored pid=43, state=RUNNABLE:DISABLE_TABLE_PREPARE; DisableTableProcedure table=TestAcidGuarantees 2023-04-15 02:55:26,907 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1681527326907"}]},"ts":"1681527326907"} 2023-04-15 02:55:26,908 INFO [PEWorker-4] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=DISABLING in hbase:meta 2023-04-15 02:55:26,910 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(1227): Checking to see if procedure is done pid=43 2023-04-15 02:55:26,914 INFO [PEWorker-4] procedure.DisableTableProcedure(293): Set TestAcidGuarantees to state=DISABLING 2023-04-15 02:55:26,915 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=44, ppid=43, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=104a1f3a98692b3503aef02f1d865942, UNASSIGN}] 2023-04-15 02:55:26,915 INFO [PEWorker-4] procedure.MasterProcedureScheduler(727): Took xlock for pid=44, ppid=43, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=104a1f3a98692b3503aef02f1d865942, UNASSIGN 2023-04-15 02:55:26,916 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=44 updating hbase:meta row=104a1f3a98692b3503aef02f1d865942, regionState=CLOSING, regionLocation=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:26,916 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1681527301940.104a1f3a98692b3503aef02f1d865942.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1681527326916"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1681527326916"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1681527326916"}]},"ts":"1681527326916"} 2023-04-15 02:55:26,917 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=45, ppid=44, state=RUNNABLE; CloseRegionProcedure 104a1f3a98692b3503aef02f1d865942, server=jenkins-hbase20.apache.org,34681,1681527209133}] 2023-04-15 02:55:27,011 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(1227): Checking to see if procedure is done pid=43 2023-04-15 02:55:27,068 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] handler.UnassignRegionHandler(109): Close 104a1f3a98692b3503aef02f1d865942 2023-04-15 02:55:27,068 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1604): Closing 104a1f3a98692b3503aef02f1d865942, disabling compactions & flushes 2023-04-15 02:55:27,068 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1858): waiting for 1 compactions & cache flush to complete for region TestAcidGuarantees,,1681527301940.104a1f3a98692b3503aef02f1d865942. 2023-04-15 02:55:27,131 DEBUG [RS:0;jenkins-hbase20:34681-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/104a1f3a98692b3503aef02f1d865942/.tmp/A/48897432a1c64224b118f3b2683ce046 as hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/104a1f3a98692b3503aef02f1d865942/A/48897432a1c64224b118f3b2683ce046 2023-04-15 02:55:27,139 INFO [RS:0;jenkins-hbase20:34681-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 7 (all) file(s) in 104a1f3a98692b3503aef02f1d865942/A of 104a1f3a98692b3503aef02f1d865942 into 48897432a1c64224b118f3b2683ce046(size=26.1 K), total size for store is 26.1 K. This selection was in queue for 0sec, and took 1sec to execute. 2023-04-15 02:55:27,139 DEBUG [RS:0;jenkins-hbase20:34681-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 104a1f3a98692b3503aef02f1d865942: 2023-04-15 02:55:27,139 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1858): waiting for 0 compactions & cache flush to complete for region TestAcidGuarantees,,1681527301940.104a1f3a98692b3503aef02f1d865942. 2023-04-15 02:55:27,139 INFO [RS:0;jenkins-hbase20:34681-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681527301940.104a1f3a98692b3503aef02f1d865942., storeName=104a1f3a98692b3503aef02f1d865942/A, priority=9, startTime=1681527325878; duration=1sec 2023-04-15 02:55:27,140 DEBUG [RS:0;jenkins-hbase20:34681-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-15 02:55:27,186 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=369, memsize=17.9 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/104a1f3a98692b3503aef02f1d865942/.tmp/A/b1dd2539a11d4139b717ef4a9befbe1c 2023-04-15 02:55:27,213 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(1227): Checking to see if procedure is done pid=43 2023-04-15 02:55:27,514 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(1227): Checking to see if procedure is done pid=43 2023-04-15 02:55:27,599 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=369 (bloomFilter=true), to=hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/104a1f3a98692b3503aef02f1d865942/.tmp/B/e66d4d9388a04d80b1b03bee62a0917a 2023-04-15 02:55:28,011 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=369 (bloomFilter=true), to=hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/104a1f3a98692b3503aef02f1d865942/.tmp/C/f6a5d714f5204f01afcd54eef27242f9 2023-04-15 02:55:28,015 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(1227): Checking to see if procedure is done pid=43 2023-04-15 02:55:28,017 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/104a1f3a98692b3503aef02f1d865942/.tmp/A/b1dd2539a11d4139b717ef4a9befbe1c as hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/104a1f3a98692b3503aef02f1d865942/A/b1dd2539a11d4139b717ef4a9befbe1c 2023-04-15 02:55:28,025 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/104a1f3a98692b3503aef02f1d865942/A/b1dd2539a11d4139b717ef4a9befbe1c, entries=150, sequenceid=369, filesize=25.4 K 2023-04-15 02:55:28,025 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/104a1f3a98692b3503aef02f1d865942/.tmp/B/e66d4d9388a04d80b1b03bee62a0917a as hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/104a1f3a98692b3503aef02f1d865942/B/e66d4d9388a04d80b1b03bee62a0917a 2023-04-15 02:55:28,030 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/104a1f3a98692b3503aef02f1d865942/B/e66d4d9388a04d80b1b03bee62a0917a, entries=150, sequenceid=369, filesize=11.9 K 2023-04-15 02:55:28,031 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/104a1f3a98692b3503aef02f1d865942/.tmp/C/f6a5d714f5204f01afcd54eef27242f9 as hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/104a1f3a98692b3503aef02f1d865942/C/f6a5d714f5204f01afcd54eef27242f9 2023-04-15 02:55:28,035 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/104a1f3a98692b3503aef02f1d865942/C/f6a5d714f5204f01afcd54eef27242f9, entries=150, sequenceid=369, filesize=11.9 K 2023-04-15 02:55:28,036 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=26.84 KB/27480 for 104a1f3a98692b3503aef02f1d865942 in 1678ms, sequenceid=369, compaction requested=false 2023-04-15 02:55:28,036 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 104a1f3a98692b3503aef02f1d865942: 2023-04-15 02:55:28,037 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1681527301940.104a1f3a98692b3503aef02f1d865942. 2023-04-15 02:55:28,037 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1681527301940.104a1f3a98692b3503aef02f1d865942. 2023-04-15 02:55:28,037 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1681527301940.104a1f3a98692b3503aef02f1d865942. after waiting 0 ms 2023-04-15 02:55:28,037 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1681527301940.104a1f3a98692b3503aef02f1d865942. 2023-04-15 02:55:28,037 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(2745): Flushing 104a1f3a98692b3503aef02f1d865942 3/3 column families, dataSize=26.84 KB heapSize=80.44 KB 2023-04-15 02:55:28,037 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 104a1f3a98692b3503aef02f1d865942, store=A 2023-04-15 02:55:28,037 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-15 02:55:28,037 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 104a1f3a98692b3503aef02f1d865942, store=B 2023-04-15 02:55:28,037 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-15 02:55:28,037 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 104a1f3a98692b3503aef02f1d865942, store=C 2023-04-15 02:55:28,038 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-15 02:55:28,455 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023041521252caf672e4341abd44f01ec697583 to hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023041521252caf672e4341abd44f01ec697583 2023-04-15 02:55:28,863 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=378, memsize=8.9 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/104a1f3a98692b3503aef02f1d865942/.tmp/A/9661eaa2795746c29ef8c3b2f45480ba 2023-04-15 02:55:29,017 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(1227): Checking to see if procedure is done pid=43 2023-04-15 02:55:29,280 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=378 (bloomFilter=true), to=hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/104a1f3a98692b3503aef02f1d865942/.tmp/B/dd8aecba6545493283a2acf9dd015ab2 2023-04-15 02:55:29,691 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=378 (bloomFilter=true), to=hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/104a1f3a98692b3503aef02f1d865942/.tmp/C/35feeadad4e84cb88105860c817d1b12 2023-04-15 02:55:29,695 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/104a1f3a98692b3503aef02f1d865942/.tmp/A/9661eaa2795746c29ef8c3b2f45480ba as hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/104a1f3a98692b3503aef02f1d865942/A/9661eaa2795746c29ef8c3b2f45480ba 2023-04-15 02:55:29,698 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/104a1f3a98692b3503aef02f1d865942/A/9661eaa2795746c29ef8c3b2f45480ba, entries=150, sequenceid=378, filesize=25.4 K 2023-04-15 02:55:29,699 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/104a1f3a98692b3503aef02f1d865942/.tmp/B/dd8aecba6545493283a2acf9dd015ab2 as hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/104a1f3a98692b3503aef02f1d865942/B/dd8aecba6545493283a2acf9dd015ab2 2023-04-15 02:55:29,702 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/104a1f3a98692b3503aef02f1d865942/B/dd8aecba6545493283a2acf9dd015ab2, entries=150, sequenceid=378, filesize=11.9 K 2023-04-15 02:55:29,703 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/104a1f3a98692b3503aef02f1d865942/.tmp/C/35feeadad4e84cb88105860c817d1b12 as hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/104a1f3a98692b3503aef02f1d865942/C/35feeadad4e84cb88105860c817d1b12 2023-04-15 02:55:29,706 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/104a1f3a98692b3503aef02f1d865942/C/35feeadad4e84cb88105860c817d1b12, entries=150, sequenceid=378, filesize=11.9 K 2023-04-15 02:55:29,707 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(2948): Finished flush of dataSize ~26.84 KB/27480, heapSize ~80.39 KB/82320, currentSize=0 B/0 for 104a1f3a98692b3503aef02f1d865942 in 1670ms, sequenceid=378, compaction requested=true 2023-04-15 02:55:29,724 DEBUG [StoreCloser-TestAcidGuarantees,,1681527301940.104a1f3a98692b3503aef02f1d865942.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/104a1f3a98692b3503aef02f1d865942/A/032ee30bdfac405b91c9b37d38512266, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/104a1f3a98692b3503aef02f1d865942/A/3cbb598a68f74c48bb9978ac9f332692, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/104a1f3a98692b3503aef02f1d865942/A/4872b863c3ac49819160a8ded98bed42, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/104a1f3a98692b3503aef02f1d865942/A/2baf48876e504d309b0eb53ae2597104, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/104a1f3a98692b3503aef02f1d865942/A/991d654bc57544848c53ef515055bdc8, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/104a1f3a98692b3503aef02f1d865942/A/c91da5073e754463afa07efe85663f69, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/104a1f3a98692b3503aef02f1d865942/A/93140167335e47b4970764378c84ea19, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/104a1f3a98692b3503aef02f1d865942/A/3127ee9695b9478896a251390e068af3, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/104a1f3a98692b3503aef02f1d865942/A/6efb8c0a53064a5cbeec795d2cb8faf7, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/104a1f3a98692b3503aef02f1d865942/A/ee83bff49f4a41f08854c29cf00b177d, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/104a1f3a98692b3503aef02f1d865942/A/10794015f4a84f0eb0bd35ac69adbe9e, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/104a1f3a98692b3503aef02f1d865942/A/ec5f7a40bd284b86a7878a2aaef40b61, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/104a1f3a98692b3503aef02f1d865942/A/fab12f4dfac74644915b7dd6c995f10d, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/104a1f3a98692b3503aef02f1d865942/A/33eb33ab038945ffaaa7258b29a920e3, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/104a1f3a98692b3503aef02f1d865942/A/25853bcb93364adf8c9bed5374757bf0, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/104a1f3a98692b3503aef02f1d865942/A/b9c13adc25ec4fe5b85c9fd9f11b5c24, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/104a1f3a98692b3503aef02f1d865942/A/5b535cd666c047c490051e7d46a2e5b2, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/104a1f3a98692b3503aef02f1d865942/A/f891948ce31f453d8d3475787916ed1c, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/104a1f3a98692b3503aef02f1d865942/A/eed0c6378f2847a1973423d6b01f9dba, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/104a1f3a98692b3503aef02f1d865942/A/e199d5e34d4a4eefb87d3a288bdfd58b] to archive 2023-04-15 02:55:29,724 DEBUG [StoreCloser-TestAcidGuarantees,,1681527301940.104a1f3a98692b3503aef02f1d865942.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-15 02:55:29,726 DEBUG [StoreCloser-TestAcidGuarantees,,1681527301940.104a1f3a98692b3503aef02f1d865942.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/104a1f3a98692b3503aef02f1d865942/A/032ee30bdfac405b91c9b37d38512266 to hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/archive/data/default/TestAcidGuarantees/104a1f3a98692b3503aef02f1d865942/A/032ee30bdfac405b91c9b37d38512266 2023-04-15 02:55:29,726 DEBUG [StoreCloser-TestAcidGuarantees,,1681527301940.104a1f3a98692b3503aef02f1d865942.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/104a1f3a98692b3503aef02f1d865942/A/3cbb598a68f74c48bb9978ac9f332692 to hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/archive/data/default/TestAcidGuarantees/104a1f3a98692b3503aef02f1d865942/A/3cbb598a68f74c48bb9978ac9f332692 2023-04-15 02:55:29,727 DEBUG [StoreCloser-TestAcidGuarantees,,1681527301940.104a1f3a98692b3503aef02f1d865942.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/104a1f3a98692b3503aef02f1d865942/A/4872b863c3ac49819160a8ded98bed42 to hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/archive/data/default/TestAcidGuarantees/104a1f3a98692b3503aef02f1d865942/A/4872b863c3ac49819160a8ded98bed42 2023-04-15 02:55:29,728 DEBUG [StoreCloser-TestAcidGuarantees,,1681527301940.104a1f3a98692b3503aef02f1d865942.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/104a1f3a98692b3503aef02f1d865942/A/2baf48876e504d309b0eb53ae2597104 to hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/archive/data/default/TestAcidGuarantees/104a1f3a98692b3503aef02f1d865942/A/2baf48876e504d309b0eb53ae2597104 2023-04-15 02:55:29,729 DEBUG [StoreCloser-TestAcidGuarantees,,1681527301940.104a1f3a98692b3503aef02f1d865942.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/104a1f3a98692b3503aef02f1d865942/A/991d654bc57544848c53ef515055bdc8 to hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/archive/data/default/TestAcidGuarantees/104a1f3a98692b3503aef02f1d865942/A/991d654bc57544848c53ef515055bdc8 2023-04-15 02:55:29,729 DEBUG [StoreCloser-TestAcidGuarantees,,1681527301940.104a1f3a98692b3503aef02f1d865942.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/104a1f3a98692b3503aef02f1d865942/A/c91da5073e754463afa07efe85663f69 to hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/archive/data/default/TestAcidGuarantees/104a1f3a98692b3503aef02f1d865942/A/c91da5073e754463afa07efe85663f69 2023-04-15 02:55:29,730 DEBUG [StoreCloser-TestAcidGuarantees,,1681527301940.104a1f3a98692b3503aef02f1d865942.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/104a1f3a98692b3503aef02f1d865942/A/93140167335e47b4970764378c84ea19 to hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/archive/data/default/TestAcidGuarantees/104a1f3a98692b3503aef02f1d865942/A/93140167335e47b4970764378c84ea19 2023-04-15 02:55:29,731 DEBUG [StoreCloser-TestAcidGuarantees,,1681527301940.104a1f3a98692b3503aef02f1d865942.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/104a1f3a98692b3503aef02f1d865942/A/3127ee9695b9478896a251390e068af3 to hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/archive/data/default/TestAcidGuarantees/104a1f3a98692b3503aef02f1d865942/A/3127ee9695b9478896a251390e068af3 2023-04-15 02:55:29,732 DEBUG [StoreCloser-TestAcidGuarantees,,1681527301940.104a1f3a98692b3503aef02f1d865942.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/104a1f3a98692b3503aef02f1d865942/A/6efb8c0a53064a5cbeec795d2cb8faf7 to hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/archive/data/default/TestAcidGuarantees/104a1f3a98692b3503aef02f1d865942/A/6efb8c0a53064a5cbeec795d2cb8faf7 2023-04-15 02:55:29,733 DEBUG [StoreCloser-TestAcidGuarantees,,1681527301940.104a1f3a98692b3503aef02f1d865942.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/104a1f3a98692b3503aef02f1d865942/A/ee83bff49f4a41f08854c29cf00b177d to hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/archive/data/default/TestAcidGuarantees/104a1f3a98692b3503aef02f1d865942/A/ee83bff49f4a41f08854c29cf00b177d 2023-04-15 02:55:29,734 DEBUG [StoreCloser-TestAcidGuarantees,,1681527301940.104a1f3a98692b3503aef02f1d865942.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/104a1f3a98692b3503aef02f1d865942/A/10794015f4a84f0eb0bd35ac69adbe9e to hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/archive/data/default/TestAcidGuarantees/104a1f3a98692b3503aef02f1d865942/A/10794015f4a84f0eb0bd35ac69adbe9e 2023-04-15 02:55:29,735 DEBUG [StoreCloser-TestAcidGuarantees,,1681527301940.104a1f3a98692b3503aef02f1d865942.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/104a1f3a98692b3503aef02f1d865942/A/ec5f7a40bd284b86a7878a2aaef40b61 to hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/archive/data/default/TestAcidGuarantees/104a1f3a98692b3503aef02f1d865942/A/ec5f7a40bd284b86a7878a2aaef40b61 2023-04-15 02:55:29,736 DEBUG [StoreCloser-TestAcidGuarantees,,1681527301940.104a1f3a98692b3503aef02f1d865942.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/104a1f3a98692b3503aef02f1d865942/A/fab12f4dfac74644915b7dd6c995f10d to hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/archive/data/default/TestAcidGuarantees/104a1f3a98692b3503aef02f1d865942/A/fab12f4dfac74644915b7dd6c995f10d 2023-04-15 02:55:29,737 DEBUG [StoreCloser-TestAcidGuarantees,,1681527301940.104a1f3a98692b3503aef02f1d865942.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/104a1f3a98692b3503aef02f1d865942/A/33eb33ab038945ffaaa7258b29a920e3 to hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/archive/data/default/TestAcidGuarantees/104a1f3a98692b3503aef02f1d865942/A/33eb33ab038945ffaaa7258b29a920e3 2023-04-15 02:55:29,739 DEBUG [StoreCloser-TestAcidGuarantees,,1681527301940.104a1f3a98692b3503aef02f1d865942.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/104a1f3a98692b3503aef02f1d865942/A/25853bcb93364adf8c9bed5374757bf0 to hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/archive/data/default/TestAcidGuarantees/104a1f3a98692b3503aef02f1d865942/A/25853bcb93364adf8c9bed5374757bf0 2023-04-15 02:55:29,740 DEBUG [StoreCloser-TestAcidGuarantees,,1681527301940.104a1f3a98692b3503aef02f1d865942.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/104a1f3a98692b3503aef02f1d865942/A/b9c13adc25ec4fe5b85c9fd9f11b5c24 to hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/archive/data/default/TestAcidGuarantees/104a1f3a98692b3503aef02f1d865942/A/b9c13adc25ec4fe5b85c9fd9f11b5c24 2023-04-15 02:55:29,741 DEBUG [StoreCloser-TestAcidGuarantees,,1681527301940.104a1f3a98692b3503aef02f1d865942.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/104a1f3a98692b3503aef02f1d865942/A/5b535cd666c047c490051e7d46a2e5b2 to hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/archive/data/default/TestAcidGuarantees/104a1f3a98692b3503aef02f1d865942/A/5b535cd666c047c490051e7d46a2e5b2 2023-04-15 02:55:29,742 DEBUG [StoreCloser-TestAcidGuarantees,,1681527301940.104a1f3a98692b3503aef02f1d865942.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/104a1f3a98692b3503aef02f1d865942/A/f891948ce31f453d8d3475787916ed1c to hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/archive/data/default/TestAcidGuarantees/104a1f3a98692b3503aef02f1d865942/A/f891948ce31f453d8d3475787916ed1c 2023-04-15 02:55:29,743 DEBUG [StoreCloser-TestAcidGuarantees,,1681527301940.104a1f3a98692b3503aef02f1d865942.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/104a1f3a98692b3503aef02f1d865942/A/eed0c6378f2847a1973423d6b01f9dba to hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/archive/data/default/TestAcidGuarantees/104a1f3a98692b3503aef02f1d865942/A/eed0c6378f2847a1973423d6b01f9dba 2023-04-15 02:55:29,744 DEBUG [StoreCloser-TestAcidGuarantees,,1681527301940.104a1f3a98692b3503aef02f1d865942.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/104a1f3a98692b3503aef02f1d865942/A/e199d5e34d4a4eefb87d3a288bdfd58b to hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/archive/data/default/TestAcidGuarantees/104a1f3a98692b3503aef02f1d865942/A/e199d5e34d4a4eefb87d3a288bdfd58b 2023-04-15 02:55:29,747 DEBUG [StoreCloser-TestAcidGuarantees,,1681527301940.104a1f3a98692b3503aef02f1d865942.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/104a1f3a98692b3503aef02f1d865942/B/697475e420914a2d85b9414f5bdfcb6e, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/104a1f3a98692b3503aef02f1d865942/B/ae3bbfafcdc747f996d9c62a7b80d13f, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/104a1f3a98692b3503aef02f1d865942/B/a62cf6cb551c4811ad937d6d3921e413, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/104a1f3a98692b3503aef02f1d865942/B/356dd4a5b0544433b64c47453bd94424, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/104a1f3a98692b3503aef02f1d865942/B/7d51e0aeb60340a3b6351c7a7e06866e, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/104a1f3a98692b3503aef02f1d865942/B/f5c1bacbe88140468f37d7341a61ca1d, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/104a1f3a98692b3503aef02f1d865942/B/8f11a364308f4fe0a469630c6f70f2a3, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/104a1f3a98692b3503aef02f1d865942/B/ed4f08d05dca469d86fa7767faadf370, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/104a1f3a98692b3503aef02f1d865942/B/aad59967db1a4dcd8841e8dca8885b2a, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/104a1f3a98692b3503aef02f1d865942/B/0b220438b5504a3b93e3041e42f9f358, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/104a1f3a98692b3503aef02f1d865942/B/f5361327d59449549ed8db22f4e18551, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/104a1f3a98692b3503aef02f1d865942/B/8a147cdb74ac49999df355b4aba62ddb, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/104a1f3a98692b3503aef02f1d865942/B/96e208c581044c5aaff7fe1378581a21, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/104a1f3a98692b3503aef02f1d865942/B/8b351a9d1f4740429641269e5aaf1ba4, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/104a1f3a98692b3503aef02f1d865942/B/6590adc88db04832ad8a64a343777501, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/104a1f3a98692b3503aef02f1d865942/B/3551e48afafc4d49a049bfee2388e032, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/104a1f3a98692b3503aef02f1d865942/B/545eb950cd234d8db5b32be8dfbb9a99, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/104a1f3a98692b3503aef02f1d865942/B/343a248dec4643f4b11379095dd65595, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/104a1f3a98692b3503aef02f1d865942/B/cac7506f16b2408b80ecf02f1d522752, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/104a1f3a98692b3503aef02f1d865942/B/dea8ed53bbc34af9bcd3d8a21ada58c4] to archive 2023-04-15 02:55:29,748 DEBUG [StoreCloser-TestAcidGuarantees,,1681527301940.104a1f3a98692b3503aef02f1d865942.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-15 02:55:29,750 DEBUG [StoreCloser-TestAcidGuarantees,,1681527301940.104a1f3a98692b3503aef02f1d865942.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/104a1f3a98692b3503aef02f1d865942/B/697475e420914a2d85b9414f5bdfcb6e to hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/archive/data/default/TestAcidGuarantees/104a1f3a98692b3503aef02f1d865942/B/697475e420914a2d85b9414f5bdfcb6e 2023-04-15 02:55:29,751 DEBUG [StoreCloser-TestAcidGuarantees,,1681527301940.104a1f3a98692b3503aef02f1d865942.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/104a1f3a98692b3503aef02f1d865942/B/ae3bbfafcdc747f996d9c62a7b80d13f to hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/archive/data/default/TestAcidGuarantees/104a1f3a98692b3503aef02f1d865942/B/ae3bbfafcdc747f996d9c62a7b80d13f 2023-04-15 02:55:29,753 DEBUG [StoreCloser-TestAcidGuarantees,,1681527301940.104a1f3a98692b3503aef02f1d865942.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/104a1f3a98692b3503aef02f1d865942/B/a62cf6cb551c4811ad937d6d3921e413 to hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/archive/data/default/TestAcidGuarantees/104a1f3a98692b3503aef02f1d865942/B/a62cf6cb551c4811ad937d6d3921e413 2023-04-15 02:55:29,754 DEBUG [StoreCloser-TestAcidGuarantees,,1681527301940.104a1f3a98692b3503aef02f1d865942.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/104a1f3a98692b3503aef02f1d865942/B/356dd4a5b0544433b64c47453bd94424 to hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/archive/data/default/TestAcidGuarantees/104a1f3a98692b3503aef02f1d865942/B/356dd4a5b0544433b64c47453bd94424 2023-04-15 02:55:29,755 DEBUG [StoreCloser-TestAcidGuarantees,,1681527301940.104a1f3a98692b3503aef02f1d865942.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/104a1f3a98692b3503aef02f1d865942/B/7d51e0aeb60340a3b6351c7a7e06866e to hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/archive/data/default/TestAcidGuarantees/104a1f3a98692b3503aef02f1d865942/B/7d51e0aeb60340a3b6351c7a7e06866e 2023-04-15 02:55:29,756 DEBUG [StoreCloser-TestAcidGuarantees,,1681527301940.104a1f3a98692b3503aef02f1d865942.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/104a1f3a98692b3503aef02f1d865942/B/f5c1bacbe88140468f37d7341a61ca1d to hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/archive/data/default/TestAcidGuarantees/104a1f3a98692b3503aef02f1d865942/B/f5c1bacbe88140468f37d7341a61ca1d 2023-04-15 02:55:29,756 DEBUG [StoreCloser-TestAcidGuarantees,,1681527301940.104a1f3a98692b3503aef02f1d865942.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/104a1f3a98692b3503aef02f1d865942/B/8f11a364308f4fe0a469630c6f70f2a3 to hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/archive/data/default/TestAcidGuarantees/104a1f3a98692b3503aef02f1d865942/B/8f11a364308f4fe0a469630c6f70f2a3 2023-04-15 02:55:29,757 DEBUG [StoreCloser-TestAcidGuarantees,,1681527301940.104a1f3a98692b3503aef02f1d865942.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/104a1f3a98692b3503aef02f1d865942/B/ed4f08d05dca469d86fa7767faadf370 to hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/archive/data/default/TestAcidGuarantees/104a1f3a98692b3503aef02f1d865942/B/ed4f08d05dca469d86fa7767faadf370 2023-04-15 02:55:29,758 DEBUG [StoreCloser-TestAcidGuarantees,,1681527301940.104a1f3a98692b3503aef02f1d865942.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/104a1f3a98692b3503aef02f1d865942/B/aad59967db1a4dcd8841e8dca8885b2a to hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/archive/data/default/TestAcidGuarantees/104a1f3a98692b3503aef02f1d865942/B/aad59967db1a4dcd8841e8dca8885b2a 2023-04-15 02:55:29,759 DEBUG [StoreCloser-TestAcidGuarantees,,1681527301940.104a1f3a98692b3503aef02f1d865942.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/104a1f3a98692b3503aef02f1d865942/B/0b220438b5504a3b93e3041e42f9f358 to hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/archive/data/default/TestAcidGuarantees/104a1f3a98692b3503aef02f1d865942/B/0b220438b5504a3b93e3041e42f9f358 2023-04-15 02:55:29,760 DEBUG [StoreCloser-TestAcidGuarantees,,1681527301940.104a1f3a98692b3503aef02f1d865942.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/104a1f3a98692b3503aef02f1d865942/B/f5361327d59449549ed8db22f4e18551 to hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/archive/data/default/TestAcidGuarantees/104a1f3a98692b3503aef02f1d865942/B/f5361327d59449549ed8db22f4e18551 2023-04-15 02:55:29,760 DEBUG [StoreCloser-TestAcidGuarantees,,1681527301940.104a1f3a98692b3503aef02f1d865942.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/104a1f3a98692b3503aef02f1d865942/B/8a147cdb74ac49999df355b4aba62ddb to hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/archive/data/default/TestAcidGuarantees/104a1f3a98692b3503aef02f1d865942/B/8a147cdb74ac49999df355b4aba62ddb 2023-04-15 02:55:29,761 DEBUG [StoreCloser-TestAcidGuarantees,,1681527301940.104a1f3a98692b3503aef02f1d865942.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/104a1f3a98692b3503aef02f1d865942/B/96e208c581044c5aaff7fe1378581a21 to hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/archive/data/default/TestAcidGuarantees/104a1f3a98692b3503aef02f1d865942/B/96e208c581044c5aaff7fe1378581a21 2023-04-15 02:55:29,762 DEBUG [StoreCloser-TestAcidGuarantees,,1681527301940.104a1f3a98692b3503aef02f1d865942.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/104a1f3a98692b3503aef02f1d865942/B/8b351a9d1f4740429641269e5aaf1ba4 to hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/archive/data/default/TestAcidGuarantees/104a1f3a98692b3503aef02f1d865942/B/8b351a9d1f4740429641269e5aaf1ba4 2023-04-15 02:55:29,763 DEBUG [StoreCloser-TestAcidGuarantees,,1681527301940.104a1f3a98692b3503aef02f1d865942.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/104a1f3a98692b3503aef02f1d865942/B/6590adc88db04832ad8a64a343777501 to hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/archive/data/default/TestAcidGuarantees/104a1f3a98692b3503aef02f1d865942/B/6590adc88db04832ad8a64a343777501 2023-04-15 02:55:29,763 DEBUG [StoreCloser-TestAcidGuarantees,,1681527301940.104a1f3a98692b3503aef02f1d865942.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/104a1f3a98692b3503aef02f1d865942/B/3551e48afafc4d49a049bfee2388e032 to hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/archive/data/default/TestAcidGuarantees/104a1f3a98692b3503aef02f1d865942/B/3551e48afafc4d49a049bfee2388e032 2023-04-15 02:55:29,764 DEBUG [StoreCloser-TestAcidGuarantees,,1681527301940.104a1f3a98692b3503aef02f1d865942.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/104a1f3a98692b3503aef02f1d865942/B/545eb950cd234d8db5b32be8dfbb9a99 to hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/archive/data/default/TestAcidGuarantees/104a1f3a98692b3503aef02f1d865942/B/545eb950cd234d8db5b32be8dfbb9a99 2023-04-15 02:55:29,765 DEBUG [StoreCloser-TestAcidGuarantees,,1681527301940.104a1f3a98692b3503aef02f1d865942.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/104a1f3a98692b3503aef02f1d865942/B/343a248dec4643f4b11379095dd65595 to hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/archive/data/default/TestAcidGuarantees/104a1f3a98692b3503aef02f1d865942/B/343a248dec4643f4b11379095dd65595 2023-04-15 02:55:29,765 DEBUG [StoreCloser-TestAcidGuarantees,,1681527301940.104a1f3a98692b3503aef02f1d865942.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/104a1f3a98692b3503aef02f1d865942/B/cac7506f16b2408b80ecf02f1d522752 to hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/archive/data/default/TestAcidGuarantees/104a1f3a98692b3503aef02f1d865942/B/cac7506f16b2408b80ecf02f1d522752 2023-04-15 02:55:29,766 DEBUG [StoreCloser-TestAcidGuarantees,,1681527301940.104a1f3a98692b3503aef02f1d865942.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/104a1f3a98692b3503aef02f1d865942/B/dea8ed53bbc34af9bcd3d8a21ada58c4 to hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/archive/data/default/TestAcidGuarantees/104a1f3a98692b3503aef02f1d865942/B/dea8ed53bbc34af9bcd3d8a21ada58c4 2023-04-15 02:55:29,769 DEBUG [StoreCloser-TestAcidGuarantees,,1681527301940.104a1f3a98692b3503aef02f1d865942.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/104a1f3a98692b3503aef02f1d865942/C/c48c0a175cd04484a3922fce0950b506, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/104a1f3a98692b3503aef02f1d865942/C/93e4377e68b9479f839f5be916fb3d7f, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/104a1f3a98692b3503aef02f1d865942/C/355d44d68c824843ad7c1f9e8f03cfe4, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/104a1f3a98692b3503aef02f1d865942/C/2a0f88f61e6c452bbd5e70ca0b1ac888, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/104a1f3a98692b3503aef02f1d865942/C/520d53b28c344c838f7c4769d039de8c, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/104a1f3a98692b3503aef02f1d865942/C/ad3099311aa949b68b3948d71019ee48, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/104a1f3a98692b3503aef02f1d865942/C/5cb02c2a67b645b8b76212926ac890f2, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/104a1f3a98692b3503aef02f1d865942/C/fa4a013e158245edab7ec80baac43f81, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/104a1f3a98692b3503aef02f1d865942/C/62fe15263bef44aea7b83db93c46c4f9, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/104a1f3a98692b3503aef02f1d865942/C/8cf30cb2449c4d4c915a6fed58dbb6cb, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/104a1f3a98692b3503aef02f1d865942/C/4502e62cc1574f8bb74a3859a770227c, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/104a1f3a98692b3503aef02f1d865942/C/75d1f54082ba4e1db8a180ae08f01d0b, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/104a1f3a98692b3503aef02f1d865942/C/44c32b452a754cc695579b554f2cddc9, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/104a1f3a98692b3503aef02f1d865942/C/486f243a5a8a47bfb55d6c75e878658d, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/104a1f3a98692b3503aef02f1d865942/C/dc2f567be0b342b3b6bf7a4e71125802, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/104a1f3a98692b3503aef02f1d865942/C/41866252f56145e09d7441da4e316361, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/104a1f3a98692b3503aef02f1d865942/C/0ce75ec5e15543de8052b6a6b4e68ce1, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/104a1f3a98692b3503aef02f1d865942/C/2005670e61c1453484cab603e00c404c, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/104a1f3a98692b3503aef02f1d865942/C/69e313b055a240f6828f4ebb73ac2c43, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/104a1f3a98692b3503aef02f1d865942/C/007d8f3a743146b18a51a5a03aa59d2c] to archive 2023-04-15 02:55:29,769 DEBUG [StoreCloser-TestAcidGuarantees,,1681527301940.104a1f3a98692b3503aef02f1d865942.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-15 02:55:29,770 DEBUG [StoreCloser-TestAcidGuarantees,,1681527301940.104a1f3a98692b3503aef02f1d865942.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/104a1f3a98692b3503aef02f1d865942/C/c48c0a175cd04484a3922fce0950b506 to hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/archive/data/default/TestAcidGuarantees/104a1f3a98692b3503aef02f1d865942/C/c48c0a175cd04484a3922fce0950b506 2023-04-15 02:55:29,771 DEBUG [StoreCloser-TestAcidGuarantees,,1681527301940.104a1f3a98692b3503aef02f1d865942.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/104a1f3a98692b3503aef02f1d865942/C/93e4377e68b9479f839f5be916fb3d7f to hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/archive/data/default/TestAcidGuarantees/104a1f3a98692b3503aef02f1d865942/C/93e4377e68b9479f839f5be916fb3d7f 2023-04-15 02:55:29,772 DEBUG [StoreCloser-TestAcidGuarantees,,1681527301940.104a1f3a98692b3503aef02f1d865942.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/104a1f3a98692b3503aef02f1d865942/C/355d44d68c824843ad7c1f9e8f03cfe4 to hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/archive/data/default/TestAcidGuarantees/104a1f3a98692b3503aef02f1d865942/C/355d44d68c824843ad7c1f9e8f03cfe4 2023-04-15 02:55:29,773 DEBUG [StoreCloser-TestAcidGuarantees,,1681527301940.104a1f3a98692b3503aef02f1d865942.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/104a1f3a98692b3503aef02f1d865942/C/2a0f88f61e6c452bbd5e70ca0b1ac888 to hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/archive/data/default/TestAcidGuarantees/104a1f3a98692b3503aef02f1d865942/C/2a0f88f61e6c452bbd5e70ca0b1ac888 2023-04-15 02:55:29,774 DEBUG [StoreCloser-TestAcidGuarantees,,1681527301940.104a1f3a98692b3503aef02f1d865942.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/104a1f3a98692b3503aef02f1d865942/C/520d53b28c344c838f7c4769d039de8c to hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/archive/data/default/TestAcidGuarantees/104a1f3a98692b3503aef02f1d865942/C/520d53b28c344c838f7c4769d039de8c 2023-04-15 02:55:29,775 DEBUG [StoreCloser-TestAcidGuarantees,,1681527301940.104a1f3a98692b3503aef02f1d865942.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/104a1f3a98692b3503aef02f1d865942/C/ad3099311aa949b68b3948d71019ee48 to hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/archive/data/default/TestAcidGuarantees/104a1f3a98692b3503aef02f1d865942/C/ad3099311aa949b68b3948d71019ee48 2023-04-15 02:55:29,776 DEBUG [StoreCloser-TestAcidGuarantees,,1681527301940.104a1f3a98692b3503aef02f1d865942.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/104a1f3a98692b3503aef02f1d865942/C/5cb02c2a67b645b8b76212926ac890f2 to hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/archive/data/default/TestAcidGuarantees/104a1f3a98692b3503aef02f1d865942/C/5cb02c2a67b645b8b76212926ac890f2 2023-04-15 02:55:29,777 DEBUG [StoreCloser-TestAcidGuarantees,,1681527301940.104a1f3a98692b3503aef02f1d865942.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/104a1f3a98692b3503aef02f1d865942/C/fa4a013e158245edab7ec80baac43f81 to hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/archive/data/default/TestAcidGuarantees/104a1f3a98692b3503aef02f1d865942/C/fa4a013e158245edab7ec80baac43f81 2023-04-15 02:55:29,778 DEBUG [StoreCloser-TestAcidGuarantees,,1681527301940.104a1f3a98692b3503aef02f1d865942.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/104a1f3a98692b3503aef02f1d865942/C/62fe15263bef44aea7b83db93c46c4f9 to hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/archive/data/default/TestAcidGuarantees/104a1f3a98692b3503aef02f1d865942/C/62fe15263bef44aea7b83db93c46c4f9 2023-04-15 02:55:29,779 DEBUG [StoreCloser-TestAcidGuarantees,,1681527301940.104a1f3a98692b3503aef02f1d865942.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/104a1f3a98692b3503aef02f1d865942/C/8cf30cb2449c4d4c915a6fed58dbb6cb to hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/archive/data/default/TestAcidGuarantees/104a1f3a98692b3503aef02f1d865942/C/8cf30cb2449c4d4c915a6fed58dbb6cb 2023-04-15 02:55:29,780 DEBUG [StoreCloser-TestAcidGuarantees,,1681527301940.104a1f3a98692b3503aef02f1d865942.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/104a1f3a98692b3503aef02f1d865942/C/4502e62cc1574f8bb74a3859a770227c to hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/archive/data/default/TestAcidGuarantees/104a1f3a98692b3503aef02f1d865942/C/4502e62cc1574f8bb74a3859a770227c 2023-04-15 02:55:29,781 DEBUG [StoreCloser-TestAcidGuarantees,,1681527301940.104a1f3a98692b3503aef02f1d865942.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/104a1f3a98692b3503aef02f1d865942/C/75d1f54082ba4e1db8a180ae08f01d0b to hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/archive/data/default/TestAcidGuarantees/104a1f3a98692b3503aef02f1d865942/C/75d1f54082ba4e1db8a180ae08f01d0b 2023-04-15 02:55:29,783 DEBUG [StoreCloser-TestAcidGuarantees,,1681527301940.104a1f3a98692b3503aef02f1d865942.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/104a1f3a98692b3503aef02f1d865942/C/44c32b452a754cc695579b554f2cddc9 to hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/archive/data/default/TestAcidGuarantees/104a1f3a98692b3503aef02f1d865942/C/44c32b452a754cc695579b554f2cddc9 2023-04-15 02:55:29,784 DEBUG [StoreCloser-TestAcidGuarantees,,1681527301940.104a1f3a98692b3503aef02f1d865942.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/104a1f3a98692b3503aef02f1d865942/C/486f243a5a8a47bfb55d6c75e878658d to hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/archive/data/default/TestAcidGuarantees/104a1f3a98692b3503aef02f1d865942/C/486f243a5a8a47bfb55d6c75e878658d 2023-04-15 02:55:29,785 DEBUG [StoreCloser-TestAcidGuarantees,,1681527301940.104a1f3a98692b3503aef02f1d865942.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/104a1f3a98692b3503aef02f1d865942/C/dc2f567be0b342b3b6bf7a4e71125802 to hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/archive/data/default/TestAcidGuarantees/104a1f3a98692b3503aef02f1d865942/C/dc2f567be0b342b3b6bf7a4e71125802 2023-04-15 02:55:29,787 DEBUG [StoreCloser-TestAcidGuarantees,,1681527301940.104a1f3a98692b3503aef02f1d865942.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/104a1f3a98692b3503aef02f1d865942/C/41866252f56145e09d7441da4e316361 to hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/archive/data/default/TestAcidGuarantees/104a1f3a98692b3503aef02f1d865942/C/41866252f56145e09d7441da4e316361 2023-04-15 02:55:29,788 DEBUG [StoreCloser-TestAcidGuarantees,,1681527301940.104a1f3a98692b3503aef02f1d865942.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/104a1f3a98692b3503aef02f1d865942/C/0ce75ec5e15543de8052b6a6b4e68ce1 to hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/archive/data/default/TestAcidGuarantees/104a1f3a98692b3503aef02f1d865942/C/0ce75ec5e15543de8052b6a6b4e68ce1 2023-04-15 02:55:29,789 DEBUG [StoreCloser-TestAcidGuarantees,,1681527301940.104a1f3a98692b3503aef02f1d865942.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/104a1f3a98692b3503aef02f1d865942/C/2005670e61c1453484cab603e00c404c to hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/archive/data/default/TestAcidGuarantees/104a1f3a98692b3503aef02f1d865942/C/2005670e61c1453484cab603e00c404c 2023-04-15 02:55:29,790 DEBUG [StoreCloser-TestAcidGuarantees,,1681527301940.104a1f3a98692b3503aef02f1d865942.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/104a1f3a98692b3503aef02f1d865942/C/69e313b055a240f6828f4ebb73ac2c43 to hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/archive/data/default/TestAcidGuarantees/104a1f3a98692b3503aef02f1d865942/C/69e313b055a240f6828f4ebb73ac2c43 2023-04-15 02:55:29,792 DEBUG [StoreCloser-TestAcidGuarantees,,1681527301940.104a1f3a98692b3503aef02f1d865942.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/104a1f3a98692b3503aef02f1d865942/C/007d8f3a743146b18a51a5a03aa59d2c to hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/archive/data/default/TestAcidGuarantees/104a1f3a98692b3503aef02f1d865942/C/007d8f3a743146b18a51a5a03aa59d2c 2023-04-15 02:55:29,797 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/104a1f3a98692b3503aef02f1d865942/recovered.edits/381.seqid, newMaxSeqId=381, maxSeqId=4 2023-04-15 02:55:29,798 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1681527301940.104a1f3a98692b3503aef02f1d865942. 2023-04-15 02:55:29,798 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1558): Region close journal for 104a1f3a98692b3503aef02f1d865942: 2023-04-15 02:55:29,801 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] handler.UnassignRegionHandler(147): Closed 104a1f3a98692b3503aef02f1d865942 2023-04-15 02:55:29,801 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=44 updating hbase:meta row=104a1f3a98692b3503aef02f1d865942, regionState=CLOSED 2023-04-15 02:55:29,801 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1681527301940.104a1f3a98692b3503aef02f1d865942.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1681527329801"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1681527329801"}]},"ts":"1681527329801"} 2023-04-15 02:55:29,805 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=45, resume processing ppid=44 2023-04-15 02:55:29,805 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=45, ppid=44, state=SUCCESS; CloseRegionProcedure 104a1f3a98692b3503aef02f1d865942, server=jenkins-hbase20.apache.org,34681,1681527209133 in 2.8870 sec 2023-04-15 02:55:29,806 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=44, resume processing ppid=43 2023-04-15 02:55:29,806 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=44, ppid=43, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=104a1f3a98692b3503aef02f1d865942, UNASSIGN in 2.8900 sec 2023-04-15 02:55:29,807 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1681527329807"}]},"ts":"1681527329807"} 2023-04-15 02:55:29,808 INFO [PEWorker-3] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=DISABLED in hbase:meta 2023-04-15 02:55:29,809 INFO [PEWorker-3] procedure.DisableTableProcedure(305): Set TestAcidGuarantees to state=DISABLED 2023-04-15 02:55:29,810 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=43, state=SUCCESS; DisableTableProcedure table=TestAcidGuarantees in 2.9060 sec 2023-04-15 02:55:30,229 DEBUG [regionserver/jenkins-hbase20:0.Chore.1] throttle.PressureAwareCompactionThroughputController(103): CompactionPressure is 0.0, tune throughput to 50.00 MB/second 2023-04-15 02:55:31,019 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(1227): Checking to see if procedure is done pid=43 2023-04-15 02:55:31,020 INFO [Listener at localhost.localdomain/33693] client.HBaseAdmin$TableFuture(3541): Operation: DISABLE, Table Name: default:TestAcidGuarantees, procId: 43 completed 2023-04-15 02:55:31,021 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.HMaster$5(2228): Client=jenkins//148.251.75.209 delete TestAcidGuarantees 2023-04-15 02:55:31,022 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] procedure2.ProcedureExecutor(1029): Stored pid=46, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION; DeleteTableProcedure table=TestAcidGuarantees 2023-04-15 02:55:31,025 DEBUG [PEWorker-2] procedure.DeleteTableProcedure(101): Waiting for RIT for pid=46, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-15 02:55:31,026 DEBUG [PEWorker-2] procedure.DeleteTableProcedure(113): Deleting regions from filesystem for pid=46, state=RUNNABLE:DELETE_TABLE_CLEAR_FS_LAYOUT, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-15 02:55:31,026 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(1227): Checking to see if procedure is done pid=46 2023-04-15 02:55:31,032 DEBUG [HFileArchiver-9] backup.HFileArchiver(131): ARCHIVING hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/.tmp/data/default/TestAcidGuarantees/104a1f3a98692b3503aef02f1d865942 2023-04-15 02:55:31,036 DEBUG [HFileArchiver-9] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/.tmp/data/default/TestAcidGuarantees/104a1f3a98692b3503aef02f1d865942/A, FileablePath, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/.tmp/data/default/TestAcidGuarantees/104a1f3a98692b3503aef02f1d865942/B, FileablePath, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/.tmp/data/default/TestAcidGuarantees/104a1f3a98692b3503aef02f1d865942/C, FileablePath, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/.tmp/data/default/TestAcidGuarantees/104a1f3a98692b3503aef02f1d865942/recovered.edits] 2023-04-15 02:55:31,040 DEBUG [HFileArchiver-9] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/.tmp/data/default/TestAcidGuarantees/104a1f3a98692b3503aef02f1d865942/A/48897432a1c64224b118f3b2683ce046 to hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/archive/data/default/TestAcidGuarantees/104a1f3a98692b3503aef02f1d865942/A/48897432a1c64224b118f3b2683ce046 2023-04-15 02:55:31,042 DEBUG [HFileArchiver-9] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/.tmp/data/default/TestAcidGuarantees/104a1f3a98692b3503aef02f1d865942/A/9661eaa2795746c29ef8c3b2f45480ba to hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/archive/data/default/TestAcidGuarantees/104a1f3a98692b3503aef02f1d865942/A/9661eaa2795746c29ef8c3b2f45480ba 2023-04-15 02:55:31,043 DEBUG [HFileArchiver-9] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/.tmp/data/default/TestAcidGuarantees/104a1f3a98692b3503aef02f1d865942/A/b1dd2539a11d4139b717ef4a9befbe1c to hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/archive/data/default/TestAcidGuarantees/104a1f3a98692b3503aef02f1d865942/A/b1dd2539a11d4139b717ef4a9befbe1c 2023-04-15 02:55:31,047 DEBUG [HFileArchiver-9] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/.tmp/data/default/TestAcidGuarantees/104a1f3a98692b3503aef02f1d865942/B/1f9d45a03b974d8d9ca9d12beffd35e7 to hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/archive/data/default/TestAcidGuarantees/104a1f3a98692b3503aef02f1d865942/B/1f9d45a03b974d8d9ca9d12beffd35e7 2023-04-15 02:55:31,049 DEBUG [HFileArchiver-9] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/.tmp/data/default/TestAcidGuarantees/104a1f3a98692b3503aef02f1d865942/B/dd8aecba6545493283a2acf9dd015ab2 to hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/archive/data/default/TestAcidGuarantees/104a1f3a98692b3503aef02f1d865942/B/dd8aecba6545493283a2acf9dd015ab2 2023-04-15 02:55:31,051 DEBUG [HFileArchiver-9] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/.tmp/data/default/TestAcidGuarantees/104a1f3a98692b3503aef02f1d865942/B/e66d4d9388a04d80b1b03bee62a0917a to hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/archive/data/default/TestAcidGuarantees/104a1f3a98692b3503aef02f1d865942/B/e66d4d9388a04d80b1b03bee62a0917a 2023-04-15 02:55:31,056 DEBUG [HFileArchiver-9] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/.tmp/data/default/TestAcidGuarantees/104a1f3a98692b3503aef02f1d865942/C/35feeadad4e84cb88105860c817d1b12 to hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/archive/data/default/TestAcidGuarantees/104a1f3a98692b3503aef02f1d865942/C/35feeadad4e84cb88105860c817d1b12 2023-04-15 02:55:31,058 DEBUG [HFileArchiver-9] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/.tmp/data/default/TestAcidGuarantees/104a1f3a98692b3503aef02f1d865942/C/d79e25fb2453479d810a799f1c72d6ff to hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/archive/data/default/TestAcidGuarantees/104a1f3a98692b3503aef02f1d865942/C/d79e25fb2453479d810a799f1c72d6ff 2023-04-15 02:55:31,060 DEBUG [HFileArchiver-9] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/.tmp/data/default/TestAcidGuarantees/104a1f3a98692b3503aef02f1d865942/C/f6a5d714f5204f01afcd54eef27242f9 to hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/archive/data/default/TestAcidGuarantees/104a1f3a98692b3503aef02f1d865942/C/f6a5d714f5204f01afcd54eef27242f9 2023-04-15 02:55:31,062 DEBUG [HFileArchiver-9] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/.tmp/data/default/TestAcidGuarantees/104a1f3a98692b3503aef02f1d865942/recovered.edits/381.seqid to hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/archive/data/default/TestAcidGuarantees/104a1f3a98692b3503aef02f1d865942/recovered.edits/381.seqid 2023-04-15 02:55:31,063 DEBUG [HFileArchiver-9] backup.HFileArchiver(596): Deleted hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/.tmp/data/default/TestAcidGuarantees/104a1f3a98692b3503aef02f1d865942 2023-04-15 02:55:31,063 DEBUG [PEWorker-2] procedure.DeleteTableProcedure(328): Archived TestAcidGuarantees regions 2023-04-15 02:55:31,063 DEBUG [PEWorker-2] backup.HFileArchiver(131): ARCHIVING hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3 2023-04-15 02:55:31,064 DEBUG [PEWorker-2] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A] 2023-04-15 02:55:31,067 DEBUG [PEWorker-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023041510bf33b278134f818061a3b7591a9811 to hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023041510bf33b278134f818061a3b7591a9811 2023-04-15 02:55:31,068 DEBUG [PEWorker-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023041516dbfe40ca474e9098188aadd6ddf7ca to hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023041516dbfe40ca474e9098188aadd6ddf7ca 2023-04-15 02:55:31,069 DEBUG [PEWorker-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023041521252caf672e4341abd44f01ec697583 to hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023041521252caf672e4341abd44f01ec697583 2023-04-15 02:55:31,070 DEBUG [PEWorker-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023041528c65241600346cab5b612fb8530642c to hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023041528c65241600346cab5b612fb8530642c 2023-04-15 02:55:31,071 DEBUG [PEWorker-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304152be756929c244a4b861d3dbeee4d51d3 to hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304152be756929c244a4b861d3dbeee4d51d3 2023-04-15 02:55:31,072 DEBUG [PEWorker-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230415306cff40458d437c819e48ab33271a62 to hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230415306cff40458d437c819e48ab33271a62 2023-04-15 02:55:31,073 DEBUG [PEWorker-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023041531ca4f9c67eb42958e71291b2f44727a to hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023041531ca4f9c67eb42958e71291b2f44727a 2023-04-15 02:55:31,073 DEBUG [PEWorker-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230415365171c90c824b43b1173a3ec6206609 to hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230415365171c90c824b43b1173a3ec6206609 2023-04-15 02:55:31,074 DEBUG [PEWorker-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230415410f6eeb16544616a4b518ca971f0b1d to hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230415410f6eeb16544616a4b518ca971f0b1d 2023-04-15 02:55:31,075 DEBUG [PEWorker-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304154130093084044f1484d00a9ea791bb59 to hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304154130093084044f1484d00a9ea791bb59 2023-04-15 02:55:31,076 DEBUG [PEWorker-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023041542b597abb7644266aed00baa3821db15 to hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023041542b597abb7644266aed00baa3821db15 2023-04-15 02:55:31,076 DEBUG [PEWorker-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230415562c08ea00f74d6493b762871d68dc04 to hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230415562c08ea00f74d6493b762871d68dc04 2023-04-15 02:55:31,077 DEBUG [PEWorker-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304156c84924d224848c8aa54ce25eefcac07 to hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304156c84924d224848c8aa54ce25eefcac07 2023-04-15 02:55:31,078 DEBUG [PEWorker-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304156f2e6c4142a449d785b4994773d2fcca to hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304156f2e6c4142a449d785b4994773d2fcca 2023-04-15 02:55:31,078 DEBUG [PEWorker-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230415701eb9f89293494eb7b8de4a88717a22 to hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230415701eb9f89293494eb7b8de4a88717a22 2023-04-15 02:55:31,079 DEBUG [PEWorker-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230415a97a6e901f3f41a19c66dde6ec889373 to hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230415a97a6e901f3f41a19c66dde6ec889373 2023-04-15 02:55:31,080 DEBUG [PEWorker-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230415d32d20facef14127baf84eaf7920ec6f to hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230415d32d20facef14127baf84eaf7920ec6f 2023-04-15 02:55:31,081 DEBUG [PEWorker-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230415ed72b9f131094c79bd7a1df22519caf6 to hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230415ed72b9f131094c79bd7a1df22519caf6 2023-04-15 02:55:31,081 DEBUG [PEWorker-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230415f0e979f2cca84ff9a35610e1f768de16 to hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230415f0e979f2cca84ff9a35610e1f768de16 2023-04-15 02:55:31,082 DEBUG [PEWorker-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230415f1273c1235c54def90fde1a4dc4aade0 to hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230415f1273c1235c54def90fde1a4dc4aade0 2023-04-15 02:55:31,083 DEBUG [PEWorker-2] backup.HFileArchiver(596): Deleted hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3 2023-04-15 02:55:31,085 DEBUG [PEWorker-2] procedure.DeleteTableProcedure(118): Deleting regions from META for pid=46, state=RUNNABLE:DELETE_TABLE_REMOVE_FROM_META, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-15 02:55:31,086 WARN [PEWorker-2] procedure.DeleteTableProcedure(384): Deleting some vestigial 1 rows of TestAcidGuarantees from hbase:meta 2023-04-15 02:55:31,087 DEBUG [PEWorker-2] procedure.DeleteTableProcedure(421): Removing 'TestAcidGuarantees' descriptor. 2023-04-15 02:55:31,088 DEBUG [PEWorker-2] procedure.DeleteTableProcedure(124): Deleting assignment state for pid=46, state=RUNNABLE:DELETE_TABLE_UNASSIGN_REGIONS, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-15 02:55:31,088 DEBUG [PEWorker-2] procedure.DeleteTableProcedure(411): Removing 'TestAcidGuarantees' from region states. 2023-04-15 02:55:31,088 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"TestAcidGuarantees,,1681527301940.104a1f3a98692b3503aef02f1d865942.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1681527331088"}]},"ts":"9223372036854775807"} 2023-04-15 02:55:31,089 INFO [PEWorker-2] hbase.MetaTableAccessor(1788): Deleted 1 regions from META 2023-04-15 02:55:31,089 DEBUG [PEWorker-2] hbase.MetaTableAccessor(1789): Deleted regions: [{ENCODED => 104a1f3a98692b3503aef02f1d865942, NAME => 'TestAcidGuarantees,,1681527301940.104a1f3a98692b3503aef02f1d865942.', STARTKEY => '', ENDKEY => ''}] 2023-04-15 02:55:31,090 DEBUG [PEWorker-2] procedure.DeleteTableProcedure(415): Marking 'TestAcidGuarantees' as deleted. 2023-04-15 02:55:31,090 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":0,"tag":[],"timestamp":"1681527331090"}]},"ts":"9223372036854775807"} 2023-04-15 02:55:31,091 INFO [PEWorker-2] hbase.MetaTableAccessor(1658): Deleted table TestAcidGuarantees state from META 2023-04-15 02:55:31,092 DEBUG [PEWorker-2] procedure.DeleteTableProcedure(130): Finished pid=46, state=RUNNABLE:DELETE_TABLE_POST_OPERATION, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-15 02:55:31,093 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=46, state=SUCCESS; DeleteTableProcedure table=TestAcidGuarantees in 71 msec 2023-04-15 02:55:31,128 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(1227): Checking to see if procedure is done pid=46 2023-04-15 02:55:31,128 INFO [Listener at localhost.localdomain/33693] client.HBaseAdmin$TableFuture(3541): Operation: DELETE, Table Name: default:TestAcidGuarantees, procId: 46 completed 2023-04-15 02:55:31,147 INFO [Listener at localhost.localdomain/33693] hbase.ResourceChecker(175): after: TestAcidGuaranteesWithEagerPolicy#testMobScanAtomicity Thread=228 (was 226) - Thread LEAK? -, OpenFileDescriptor=528 (was 553), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=344 (was 380), ProcessCount=168 (was 169), AvailableMemoryMB=5607 (was 5735) 2023-04-15 02:55:31,157 INFO [Listener at localhost.localdomain/33693] hbase.ResourceChecker(147): before: TestAcidGuaranteesWithEagerPolicy#testScanAtomicity Thread=228, OpenFileDescriptor=528, MaxFileDescriptor=60000, SystemLoadAverage=344, ProcessCount=168, AvailableMemoryMB=5606 2023-04-15 02:55:31,159 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] util.TableDescriptorChecker(340): MEMSTORE_FLUSHSIZE for table descriptor or "hbase.hregion.memstore.flush.size" (131072) is too small, which might cause very frequent flushing. 2023-04-15 02:55:31,159 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.HMaster$4(2112): Client=jenkins//148.251.75.209 create 'TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-04-15 02:55:31,160 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] procedure2.ProcedureExecutor(1029): Stored pid=47, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION; CreateTableProcedure table=TestAcidGuarantees 2023-04-15 02:55:31,161 INFO [PEWorker-4] procedure.CreateTableProcedure(80): pid=47, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_PRE_OPERATION 2023-04-15 02:55:31,162 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(697): Client=jenkins//148.251.75.209 procedure request for creating table: namespace: "default" qualifier: "TestAcidGuarantees" procId is: 47 2023-04-15 02:55:31,162 INFO [PEWorker-4] procedure.CreateTableProcedure(80): pid=47, state=RUNNABLE:CREATE_TABLE_WRITE_FS_LAYOUT, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_WRITE_FS_LAYOUT 2023-04-15 02:55:31,162 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(1227): Checking to see if procedure is done pid=47 2023-04-15 02:55:31,163 DEBUG [HFileArchiver-10] backup.HFileArchiver(131): ARCHIVING hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/.tmp/data/default/TestAcidGuarantees/a29fa882a9e3f50c391e3aa08c5c5e72 2023-04-15 02:55:31,163 DEBUG [HFileArchiver-10] backup.HFileArchiver(153): Directory hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/.tmp/data/default/TestAcidGuarantees/a29fa882a9e3f50c391e3aa08c5c5e72 empty. 2023-04-15 02:55:31,164 DEBUG [HFileArchiver-10] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/.tmp/data/default/TestAcidGuarantees/a29fa882a9e3f50c391e3aa08c5c5e72 2023-04-15 02:55:31,164 DEBUG [PEWorker-4] procedure.DeleteTableProcedure(328): Archived TestAcidGuarantees regions 2023-04-15 02:55:31,264 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(1227): Checking to see if procedure is done pid=47 2023-04-15 02:55:31,467 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(1227): Checking to see if procedure is done pid=47 2023-04-15 02:55:31,574 DEBUG [PEWorker-4] util.FSTableDescriptors(570): Wrote into hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/.tmp/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000001 2023-04-15 02:55:31,578 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(7675): creating {ENCODED => a29fa882a9e3f50c391e3aa08c5c5e72, NAME => 'TestAcidGuarantees,,1681527331158.a29fa882a9e3f50c391e3aa08c5c5e72.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/.tmp 2023-04-15 02:55:31,586 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1681527331158.a29fa882a9e3f50c391e3aa08c5c5e72.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-15 02:55:31,587 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1604): Closing a29fa882a9e3f50c391e3aa08c5c5e72, disabling compactions & flushes 2023-04-15 02:55:31,587 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1681527331158.a29fa882a9e3f50c391e3aa08c5c5e72. 2023-04-15 02:55:31,587 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1681527331158.a29fa882a9e3f50c391e3aa08c5c5e72. 2023-04-15 02:55:31,587 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1681527331158.a29fa882a9e3f50c391e3aa08c5c5e72. after waiting 0 ms 2023-04-15 02:55:31,587 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1681527331158.a29fa882a9e3f50c391e3aa08c5c5e72. 2023-04-15 02:55:31,587 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1681527331158.a29fa882a9e3f50c391e3aa08c5c5e72. 2023-04-15 02:55:31,587 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1558): Region close journal for a29fa882a9e3f50c391e3aa08c5c5e72: 2023-04-15 02:55:31,589 INFO [PEWorker-4] procedure.CreateTableProcedure(80): pid=47, state=RUNNABLE:CREATE_TABLE_ADD_TO_META, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_ADD_TO_META 2023-04-15 02:55:31,590 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1681527331158.a29fa882a9e3f50c391e3aa08c5c5e72.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1681527331589"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1681527331589"}]},"ts":"1681527331589"} 2023-04-15 02:55:31,590 INFO [PEWorker-4] hbase.MetaTableAccessor(1496): Added 1 regions to meta. 2023-04-15 02:55:31,591 INFO [PEWorker-4] procedure.CreateTableProcedure(80): pid=47, state=RUNNABLE:CREATE_TABLE_ASSIGN_REGIONS, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_ASSIGN_REGIONS 2023-04-15 02:55:31,591 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1681527331591"}]},"ts":"1681527331591"} 2023-04-15 02:55:31,592 INFO [PEWorker-4] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=ENABLING in hbase:meta 2023-04-15 02:55:31,594 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=48, ppid=47, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=TestAcidGuarantees, region=a29fa882a9e3f50c391e3aa08c5c5e72, ASSIGN}] 2023-04-15 02:55:31,595 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=48, ppid=47, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=TestAcidGuarantees, region=a29fa882a9e3f50c391e3aa08c5c5e72, ASSIGN 2023-04-15 02:55:31,596 INFO [PEWorker-1] assignment.TransitRegionStateProcedure(193): Starting pid=48, ppid=47, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=TestAcidGuarantees, region=a29fa882a9e3f50c391e3aa08c5c5e72, ASSIGN; state=OFFLINE, location=jenkins-hbase20.apache.org,34681,1681527209133; forceNewPlan=false, retain=false 2023-04-15 02:55:31,748 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=48 updating hbase:meta row=a29fa882a9e3f50c391e3aa08c5c5e72, regionState=OPENING, regionLocation=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:31,748 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1681527331158.a29fa882a9e3f50c391e3aa08c5c5e72.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1681527331748"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1681527331748"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1681527331748"}]},"ts":"1681527331748"} 2023-04-15 02:55:31,751 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=49, ppid=48, state=RUNNABLE; OpenRegionProcedure a29fa882a9e3f50c391e3aa08c5c5e72, server=jenkins-hbase20.apache.org,34681,1681527209133}] 2023-04-15 02:55:31,768 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(1227): Checking to see if procedure is done pid=47 2023-04-15 02:55:31,912 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] handler.AssignRegionHandler(128): Open TestAcidGuarantees,,1681527331158.a29fa882a9e3f50c391e3aa08c5c5e72. 2023-04-15 02:55:31,913 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => a29fa882a9e3f50c391e3aa08c5c5e72, NAME => 'TestAcidGuarantees,,1681527331158.a29fa882a9e3f50c391e3aa08c5c5e72.', STARTKEY => '', ENDKEY => ''} 2023-04-15 02:55:31,913 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table TestAcidGuarantees a29fa882a9e3f50c391e3aa08c5c5e72 2023-04-15 02:55:31,913 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1681527331158.a29fa882a9e3f50c391e3aa08c5c5e72.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-15 02:55:31,913 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7894): checking encryption for a29fa882a9e3f50c391e3aa08c5c5e72 2023-04-15 02:55:31,914 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7897): checking classloading for a29fa882a9e3f50c391e3aa08c5c5e72 2023-04-15 02:55:31,916 INFO [StoreOpener-a29fa882a9e3f50c391e3aa08c5c5e72-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family A of region a29fa882a9e3f50c391e3aa08c5c5e72 2023-04-15 02:55:31,918 DEBUG [StoreOpener-a29fa882a9e3f50c391e3aa08c5c5e72-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/a29fa882a9e3f50c391e3aa08c5c5e72/A 2023-04-15 02:55:31,918 DEBUG [StoreOpener-a29fa882a9e3f50c391e3aa08c5c5e72-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/a29fa882a9e3f50c391e3aa08c5c5e72/A 2023-04-15 02:55:31,918 INFO [StoreOpener-a29fa882a9e3f50c391e3aa08c5c5e72-1] regionserver.CompactingMemStore(121): Store=A, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-15 02:55:31,918 INFO [StoreOpener-a29fa882a9e3f50c391e3aa08c5c5e72-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region a29fa882a9e3f50c391e3aa08c5c5e72 columnFamilyName A 2023-04-15 02:55:31,919 INFO [StoreOpener-a29fa882a9e3f50c391e3aa08c5c5e72-1] regionserver.HStore(310): Store=a29fa882a9e3f50c391e3aa08c5c5e72/A, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-15 02:55:31,919 INFO [StoreOpener-a29fa882a9e3f50c391e3aa08c5c5e72-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family B of region a29fa882a9e3f50c391e3aa08c5c5e72 2023-04-15 02:55:31,920 DEBUG [StoreOpener-a29fa882a9e3f50c391e3aa08c5c5e72-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/a29fa882a9e3f50c391e3aa08c5c5e72/B 2023-04-15 02:55:31,920 DEBUG [StoreOpener-a29fa882a9e3f50c391e3aa08c5c5e72-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/a29fa882a9e3f50c391e3aa08c5c5e72/B 2023-04-15 02:55:31,920 INFO [StoreOpener-a29fa882a9e3f50c391e3aa08c5c5e72-1] regionserver.CompactingMemStore(121): Store=B, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-15 02:55:31,921 INFO [StoreOpener-a29fa882a9e3f50c391e3aa08c5c5e72-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region a29fa882a9e3f50c391e3aa08c5c5e72 columnFamilyName B 2023-04-15 02:55:31,921 INFO [StoreOpener-a29fa882a9e3f50c391e3aa08c5c5e72-1] regionserver.HStore(310): Store=a29fa882a9e3f50c391e3aa08c5c5e72/B, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-15 02:55:31,921 INFO [StoreOpener-a29fa882a9e3f50c391e3aa08c5c5e72-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family C of region a29fa882a9e3f50c391e3aa08c5c5e72 2023-04-15 02:55:31,923 DEBUG [StoreOpener-a29fa882a9e3f50c391e3aa08c5c5e72-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/a29fa882a9e3f50c391e3aa08c5c5e72/C 2023-04-15 02:55:31,923 DEBUG [StoreOpener-a29fa882a9e3f50c391e3aa08c5c5e72-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/a29fa882a9e3f50c391e3aa08c5c5e72/C 2023-04-15 02:55:31,923 INFO [StoreOpener-a29fa882a9e3f50c391e3aa08c5c5e72-1] regionserver.CompactingMemStore(121): Store=C, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-15 02:55:31,923 INFO [StoreOpener-a29fa882a9e3f50c391e3aa08c5c5e72-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region a29fa882a9e3f50c391e3aa08c5c5e72 columnFamilyName C 2023-04-15 02:55:31,924 INFO [StoreOpener-a29fa882a9e3f50c391e3aa08c5c5e72-1] regionserver.HStore(310): Store=a29fa882a9e3f50c391e3aa08c5c5e72/C, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-15 02:55:31,924 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1146): Setting FlushNonSloppyStoresFirstPolicy for the region=TestAcidGuarantees,,1681527331158.a29fa882a9e3f50c391e3aa08c5c5e72. 2023-04-15 02:55:31,925 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/a29fa882a9e3f50c391e3aa08c5c5e72 2023-04-15 02:55:31,925 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/a29fa882a9e3f50c391e3aa08c5c5e72 2023-04-15 02:55:31,928 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table TestAcidGuarantees descriptor;using region.getMemStoreFlushHeapSize/# of families (16.0 M)) instead. 2023-04-15 02:55:31,929 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1055): writing seq id for a29fa882a9e3f50c391e3aa08c5c5e72 2023-04-15 02:55:31,931 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/a29fa882a9e3f50c391e3aa08c5c5e72/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-04-15 02:55:31,932 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1072): Opened a29fa882a9e3f50c391e3aa08c5c5e72; next sequenceid=2; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=69544276, jitterRate=0.03629046678543091}, FlushLargeStoresPolicy{flushSizeLowerBound=16777216} 2023-04-15 02:55:31,932 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(965): Region open journal for a29fa882a9e3f50c391e3aa08c5c5e72: 2023-04-15 02:55:31,933 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for TestAcidGuarantees,,1681527331158.a29fa882a9e3f50c391e3aa08c5c5e72., pid=49, masterSystemTime=1681527331904 2023-04-15 02:55:31,934 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for TestAcidGuarantees,,1681527331158.a29fa882a9e3f50c391e3aa08c5c5e72. 2023-04-15 02:55:31,934 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] handler.AssignRegionHandler(156): Opened TestAcidGuarantees,,1681527331158.a29fa882a9e3f50c391e3aa08c5c5e72. 2023-04-15 02:55:31,935 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=48 updating hbase:meta row=a29fa882a9e3f50c391e3aa08c5c5e72, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:31,935 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"TestAcidGuarantees,,1681527331158.a29fa882a9e3f50c391e3aa08c5c5e72.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1681527331935"},{"qualifier":"server","vlen":32,"tag":[],"timestamp":"1681527331935"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1681527331935"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1681527331935"}]},"ts":"1681527331935"} 2023-04-15 02:55:31,937 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=49, resume processing ppid=48 2023-04-15 02:55:31,937 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=49, ppid=48, state=SUCCESS; OpenRegionProcedure a29fa882a9e3f50c391e3aa08c5c5e72, server=jenkins-hbase20.apache.org,34681,1681527209133 in 185 msec 2023-04-15 02:55:31,938 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=48, resume processing ppid=47 2023-04-15 02:55:31,938 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=48, ppid=47, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=a29fa882a9e3f50c391e3aa08c5c5e72, ASSIGN in 343 msec 2023-04-15 02:55:31,939 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=47, state=RUNNABLE:CREATE_TABLE_UPDATE_DESC_CACHE, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_UPDATE_DESC_CACHE 2023-04-15 02:55:31,939 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1681527331939"}]},"ts":"1681527331939"} 2023-04-15 02:55:31,939 INFO [PEWorker-1] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=ENABLED in hbase:meta 2023-04-15 02:55:31,941 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=47, state=RUNNABLE:CREATE_TABLE_POST_OPERATION, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_POST_OPERATION 2023-04-15 02:55:31,941 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=47, state=SUCCESS; CreateTableProcedure table=TestAcidGuarantees in 782 msec 2023-04-15 02:55:32,270 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(1227): Checking to see if procedure is done pid=47 2023-04-15 02:55:32,271 INFO [Listener at localhost.localdomain/33693] client.HBaseAdmin$TableFuture(3541): Operation: CREATE, Table Name: default:TestAcidGuarantees, procId: 47 completed 2023-04-15 02:55:32,272 DEBUG [Listener at localhost.localdomain/33693] zookeeper.ReadOnlyZKClient(139): Connect 0x1a0b0d8a to 127.0.0.1:51762 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-15 02:55:32,281 DEBUG [Listener at localhost.localdomain/33693] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@7d12b37a, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-15 02:55:32,282 DEBUG [Listener at localhost.localdomain/33693] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-15 02:55:32,284 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:40140, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-15 02:55:32,285 DEBUG [Listener at localhost.localdomain/33693] ipc.RpcConnection(124): Using SIMPLE authentication for service=MasterService, sasl=false 2023-04-15 02:55:32,287 INFO [RS-EventLoopGroup-1-2] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:51072, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=MasterService 2023-04-15 02:55:32,289 DEBUG [Listener at localhost.localdomain/33693] zookeeper.ReadOnlyZKClient(139): Connect 0x517cd407 to 127.0.0.1:51762 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-15 02:55:32,293 DEBUG [Listener at localhost.localdomain/33693] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@13d9e4ff, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-15 02:55:32,293 DEBUG [Listener at localhost.localdomain/33693] zookeeper.ReadOnlyZKClient(139): Connect 0x3e895a5d to 127.0.0.1:51762 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-15 02:55:32,305 DEBUG [Listener at localhost.localdomain/33693] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@9e26d67, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-15 02:55:32,305 DEBUG [Listener at localhost.localdomain/33693] zookeeper.ReadOnlyZKClient(139): Connect 0x6ea7ac79 to 127.0.0.1:51762 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-15 02:55:32,313 DEBUG [Listener at localhost.localdomain/33693] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@1a885842, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-15 02:55:32,314 DEBUG [Listener at localhost.localdomain/33693] zookeeper.ReadOnlyZKClient(139): Connect 0x51ff7cd4 to 127.0.0.1:51762 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-15 02:55:32,323 DEBUG [Listener at localhost.localdomain/33693] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@2e8c6f6, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-15 02:55:32,324 DEBUG [Listener at localhost.localdomain/33693] zookeeper.ReadOnlyZKClient(139): Connect 0x06846250 to 127.0.0.1:51762 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-15 02:55:32,332 DEBUG [Listener at localhost.localdomain/33693] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@2cb7a788, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-15 02:55:32,333 DEBUG [Listener at localhost.localdomain/33693] zookeeper.ReadOnlyZKClient(139): Connect 0x41ccedcb to 127.0.0.1:51762 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-15 02:55:32,340 DEBUG [Listener at localhost.localdomain/33693] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@50d6899d, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-15 02:55:32,340 DEBUG [Listener at localhost.localdomain/33693] zookeeper.ReadOnlyZKClient(139): Connect 0x40e6f624 to 127.0.0.1:51762 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-15 02:55:32,348 DEBUG [Listener at localhost.localdomain/33693] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@6662e946, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-15 02:55:32,348 DEBUG [Listener at localhost.localdomain/33693] zookeeper.ReadOnlyZKClient(139): Connect 0x2a5d7fcf to 127.0.0.1:51762 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-15 02:55:32,355 DEBUG [Listener at localhost.localdomain/33693] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@5d7deb76, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-15 02:55:32,355 DEBUG [Listener at localhost.localdomain/33693] zookeeper.ReadOnlyZKClient(139): Connect 0x4ea9d51a to 127.0.0.1:51762 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-15 02:55:32,365 DEBUG [Listener at localhost.localdomain/33693] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@f4b7807, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-15 02:55:32,365 DEBUG [Listener at localhost.localdomain/33693] zookeeper.ReadOnlyZKClient(139): Connect 0x31fd1c1f to 127.0.0.1:51762 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-15 02:55:32,373 DEBUG [Listener at localhost.localdomain/33693] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@31fc0e3f, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-15 02:55:32,379 DEBUG [hconnection-0x163913ee-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-15 02:55:32,379 DEBUG [hconnection-0x1fefebce-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-15 02:55:32,380 DEBUG [hconnection-0x76d03e3c-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-15 02:55:32,380 DEBUG [hconnection-0x66c93a8e-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-15 02:55:32,380 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:40158, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-15 02:55:32,380 DEBUG [hconnection-0x560a549c-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-15 02:55:32,382 DEBUG [hconnection-0xe9b7743-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-15 02:55:32,382 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:40150, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-15 02:55:32,382 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:40192, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-15 02:55:32,382 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-15 02:55:32,382 DEBUG [hconnection-0x26e28ef3-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-15 02:55:32,385 DEBUG [hconnection-0x69258fc3-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-15 02:55:32,386 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:40212, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-15 02:55:32,386 DEBUG [hconnection-0x6171a94d-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-15 02:55:32,386 DEBUG [hconnection-0x4f90ce81-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-15 02:55:32,390 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:40176, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-15 02:55:32,390 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:40166, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-15 02:55:32,391 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:40210, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-15 02:55:32,391 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:40208, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-15 02:55:32,394 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:40234, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-15 02:55:32,396 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:40238, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-15 02:55:32,398 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-15 02:55:32,399 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-15 02:55:32,399 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-15 02:55:32,399 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-15 02:55:32,399 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-15 02:55:32,399 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-15 02:55:32,401 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:32,401 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:55:32,402 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-15 02:55:32,402 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-15 02:55:32,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-15 02:55:32,402 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:32,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:55:32,403 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:32,403 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-15 02:55:32,403 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:32,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-15 02:55:32,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:55:32,403 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-15 02:55:32,403 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-15 02:55:32,403 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-15 02:55:32,403 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-15 02:55:32,403 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-15 02:55:32,403 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-15 02:55:32,404 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-15 02:55:32,404 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-15 02:55:32,404 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681527331158.a29fa882a9e3f50c391e3aa08c5c5e72. 2023-04-15 02:55:32,405 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681527331158.a29fa882a9e3f50c391e3aa08c5c5e72. started... 2023-04-15 02:55:32,406 INFO [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing a29fa882a9e3f50c391e3aa08c5c5e72 3/3 column families, dataSize=26.84 KB heapSize=80.44 KB 2023-04-15 02:55:32,406 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a29fa882a9e3f50c391e3aa08c5c5e72, store=A 2023-04-15 02:55:32,406 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-15 02:55:32,406 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a29fa882a9e3f50c391e3aa08c5c5e72, store=B 2023-04-15 02:55:32,406 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-15 02:55:32,406 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a29fa882a9e3f50c391e3aa08c5c5e72, store=C 2023-04-15 02:55:32,406 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-15 02:55:32,409 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34681] regionserver.HRegion(9158): Flush requested on a29fa882a9e3f50c391e3aa08c5c5e72 2023-04-15 02:55:32,409 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681527331158.a29fa882a9e3f50c391e3aa08c5c5e72. as already flushing 2023-04-15 02:55:32,421 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=309, cellSize=14145, totalHeapSize=32376, min timestamp=1681527332415, max timestamp=1681527332420Num uniques 859; 2023-04-15 02:55:32,421 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=14145, suffix heap size=117064,new segment heap size=32376 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=309 2023-04-15 02:55:32,421 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=309, cellSize=14145, totalHeapSize=32376, min timestamp=1681527332415, max timestamp=1681527332420Num uniques 859; 2023-04-15 02:55:32,421 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=309, cellSize=14145, totalHeapSize=32376, min timestamp=1681527332415, max timestamp=1681527332420Num uniques 859; 2023-04-15 02:55:32,421 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-9] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=14145, suffix heap size=117064,new segment heap size=32376 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=309 2023-04-15 02:55:32,422 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-5] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=14145, suffix heap size=117064,new segment heap size=32376 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=309 2023-04-15 02:55:32,427 INFO [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=9 (bloomFilter=true), to=hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/a29fa882a9e3f50c391e3aa08c5c5e72/.tmp/A/0e96c61d77dc4e7196e63b48bf12904c 2023-04-15 02:55:32,436 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a29fa882a9e3f50c391e3aa08c5c5e72, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:55:32,436 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34681] ipc.CallRunner(144): callId: 9 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:40210 deadline: 1681527392433, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a29fa882a9e3f50c391e3aa08c5c5e72, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:32,436 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a29fa882a9e3f50c391e3aa08c5c5e72, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:55:32,436 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34681] ipc.CallRunner(144): callId: 9 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:40234 deadline: 1681527392434, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a29fa882a9e3f50c391e3aa08c5c5e72, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:32,438 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a29fa882a9e3f50c391e3aa08c5c5e72, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:55:32,438 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34681] ipc.CallRunner(144): callId: 11 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:40158 deadline: 1681527392435, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a29fa882a9e3f50c391e3aa08c5c5e72, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:32,438 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a29fa882a9e3f50c391e3aa08c5c5e72, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:55:32,438 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34681] ipc.CallRunner(144): callId: 9 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:40212 deadline: 1681527392435, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a29fa882a9e3f50c391e3aa08c5c5e72, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:32,438 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a29fa882a9e3f50c391e3aa08c5c5e72, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:55:32,438 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34681] ipc.CallRunner(144): callId: 11 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:40192 deadline: 1681527392436, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a29fa882a9e3f50c391e3aa08c5c5e72, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:32,542 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a29fa882a9e3f50c391e3aa08c5c5e72, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:55:32,542 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34681] ipc.CallRunner(144): callId: 11 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:40210 deadline: 1681527392537, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a29fa882a9e3f50c391e3aa08c5c5e72, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:32,543 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a29fa882a9e3f50c391e3aa08c5c5e72, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:55:32,543 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34681] ipc.CallRunner(144): callId: 11 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:40234 deadline: 1681527392537, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a29fa882a9e3f50c391e3aa08c5c5e72, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:32,543 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a29fa882a9e3f50c391e3aa08c5c5e72, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:55:32,543 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34681] ipc.CallRunner(144): callId: 13 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:40158 deadline: 1681527392539, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a29fa882a9e3f50c391e3aa08c5c5e72, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:32,543 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a29fa882a9e3f50c391e3aa08c5c5e72, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:55:32,543 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34681] ipc.CallRunner(144): callId: 11 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:40212 deadline: 1681527392539, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a29fa882a9e3f50c391e3aa08c5c5e72, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:32,543 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a29fa882a9e3f50c391e3aa08c5c5e72, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:55:32,543 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34681] ipc.CallRunner(144): callId: 13 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:40192 deadline: 1681527392539, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a29fa882a9e3f50c391e3aa08c5c5e72, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:32,746 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a29fa882a9e3f50c391e3aa08c5c5e72, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:55:32,746 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34681] ipc.CallRunner(144): callId: 13 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:40234 deadline: 1681527392743, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a29fa882a9e3f50c391e3aa08c5c5e72, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:32,747 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a29fa882a9e3f50c391e3aa08c5c5e72, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:55:32,747 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34681] ipc.CallRunner(144): callId: 15 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:40192 deadline: 1681527392744, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a29fa882a9e3f50c391e3aa08c5c5e72, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:32,747 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a29fa882a9e3f50c391e3aa08c5c5e72, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:55:32,747 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34681] ipc.CallRunner(144): callId: 13 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:40210 deadline: 1681527392744, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a29fa882a9e3f50c391e3aa08c5c5e72, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:32,747 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a29fa882a9e3f50c391e3aa08c5c5e72, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:55:32,747 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34681] ipc.CallRunner(144): callId: 15 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:40158 deadline: 1681527392744, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a29fa882a9e3f50c391e3aa08c5c5e72, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:32,747 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a29fa882a9e3f50c391e3aa08c5c5e72, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:55:32,748 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34681] ipc.CallRunner(144): callId: 13 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:40212 deadline: 1681527392745, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a29fa882a9e3f50c391e3aa08c5c5e72, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:32,867 INFO [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=9 (bloomFilter=true), to=hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/a29fa882a9e3f50c391e3aa08c5c5e72/.tmp/B/7c8b6c7189e14c99b6df874cccccf3e4 2023-04-15 02:55:33,050 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a29fa882a9e3f50c391e3aa08c5c5e72, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:55:33,050 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34681] ipc.CallRunner(144): callId: 15 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:40234 deadline: 1681527393047, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a29fa882a9e3f50c391e3aa08c5c5e72, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:33,050 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a29fa882a9e3f50c391e3aa08c5c5e72, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:55:33,051 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34681] ipc.CallRunner(144): callId: 17 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:40192 deadline: 1681527393048, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a29fa882a9e3f50c391e3aa08c5c5e72, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:33,051 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a29fa882a9e3f50c391e3aa08c5c5e72, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:55:33,051 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34681] ipc.CallRunner(144): callId: 17 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:40158 deadline: 1681527393048, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a29fa882a9e3f50c391e3aa08c5c5e72, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:33,052 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a29fa882a9e3f50c391e3aa08c5c5e72, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:55:33,052 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34681] ipc.CallRunner(144): callId: 15 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:40210 deadline: 1681527393050, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a29fa882a9e3f50c391e3aa08c5c5e72, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:33,052 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a29fa882a9e3f50c391e3aa08c5c5e72, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:55:33,052 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34681] ipc.CallRunner(144): callId: 15 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:40212 deadline: 1681527393050, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a29fa882a9e3f50c391e3aa08c5c5e72, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:33,276 INFO [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=9 (bloomFilter=true), to=hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/a29fa882a9e3f50c391e3aa08c5c5e72/.tmp/C/4b027f62b2504671b04f87e52c4b6d59 2023-04-15 02:55:33,279 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/a29fa882a9e3f50c391e3aa08c5c5e72/.tmp/A/0e96c61d77dc4e7196e63b48bf12904c as hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/a29fa882a9e3f50c391e3aa08c5c5e72/A/0e96c61d77dc4e7196e63b48bf12904c 2023-04-15 02:55:33,285 INFO [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/a29fa882a9e3f50c391e3aa08c5c5e72/A/0e96c61d77dc4e7196e63b48bf12904c, entries=150, sequenceid=9, filesize=11.6 K 2023-04-15 02:55:33,286 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/a29fa882a9e3f50c391e3aa08c5c5e72/.tmp/B/7c8b6c7189e14c99b6df874cccccf3e4 as hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/a29fa882a9e3f50c391e3aa08c5c5e72/B/7c8b6c7189e14c99b6df874cccccf3e4 2023-04-15 02:55:33,289 INFO [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/a29fa882a9e3f50c391e3aa08c5c5e72/B/7c8b6c7189e14c99b6df874cccccf3e4, entries=150, sequenceid=9, filesize=11.6 K 2023-04-15 02:55:33,290 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/a29fa882a9e3f50c391e3aa08c5c5e72/.tmp/C/4b027f62b2504671b04f87e52c4b6d59 as hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/a29fa882a9e3f50c391e3aa08c5c5e72/C/4b027f62b2504671b04f87e52c4b6d59 2023-04-15 02:55:33,293 INFO [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/a29fa882a9e3f50c391e3aa08c5c5e72/C/4b027f62b2504671b04f87e52c4b6d59, entries=150, sequenceid=9, filesize=11.6 K 2023-04-15 02:55:33,293 INFO [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~33.54 KB/34350, heapSize ~100.31 KB/102720, currentSize=154.31 KB/158010 for a29fa882a9e3f50c391e3aa08c5c5e72 in 887ms, sequenceid=9, compaction requested=false 2023-04-15 02:55:33,294 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.MetricsTableSourceImpl(130): Creating new MetricsTableSourceImpl for table 'TestAcidGuarantees' 2023-04-15 02:55:33,294 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for a29fa882a9e3f50c391e3aa08c5c5e72: 2023-04-15 02:55:33,294 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681527331158.a29fa882a9e3f50c391e3aa08c5c5e72. 2023-04-15 02:55:33,294 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-15 02:55:33,294 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-15 02:55:33,295 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:33,295 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-15 02:55:33,295 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34681,1681527209133' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-15 02:55:33,295 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:33,295 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:33,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:33,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:55:33,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:55:33,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:55:33,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:55:33,296 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:33,296 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-15 02:55:33,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:55:33,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:33,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:55:33,297 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34681,1681527209133' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-15 02:55:33,298 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@63a01a7e[Count = 0] remaining members to acquire global barrier 2023-04-15 02:55:33,298 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-15 02:55:33,298 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:33,299 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:33,299 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-15 02:55:33,300 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:33,300 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:33,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:33,300 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-15 02:55:33,300 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-15 02:55:33,300 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34681,1681527209133' in zk 2023-04-15 02:55:33,300 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:33,301 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-15 02:55:33,301 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-15 02:55:33,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:33,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:55:33,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:55:33,301 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-15 02:55:33,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:55:33,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:55:33,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:55:33,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:33,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:55:33,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:55:33,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:33,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34681,1681527209133': 2023-04-15 02:55:33,302 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34681,1681527209133' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-15 02:55:33,302 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-15 02:55:33,302 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-15 02:55:33,302 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-15 02:55:33,302 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-15 02:55:33,302 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-15 02:55:33,303 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:33,303 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:33,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:33,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:55:33,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:55:33,303 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:33,303 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-15 02:55:33,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:33,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:55:33,303 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:33,303 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-15 02:55:33,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-15 02:55:33,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:55:33,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:33,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:55:33,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:55:33,306 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:33,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:33,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:55:33,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:55:33,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:33,307 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:33,307 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-15 02:55:33,307 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:55:33,307 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:55:33,307 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:33,307 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-15 02:55:33,307 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-15 02:55:33,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-15 02:55:33,307 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-15 02:55:33,307 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-15 02:55:33,307 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-15 02:55:33,307 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:33,307 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-15 02:55:33,307 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:33,307 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:33,307 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-15 02:55:33,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-15 02:55:33,308 DEBUG [Thread-2739] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-15 02:55:33,308 DEBUG [Thread-2739] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-15 02:55:33,408 DEBUG [Thread-2739] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-15 02:55:33,408 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-15 02:55:33,410 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-15 02:55:33,411 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-15 02:55:33,411 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-15 02:55:33,411 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-15 02:55:33,411 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-15 02:55:33,411 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-15 02:55:33,411 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-15 02:55:33,411 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:33,411 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:55:33,412 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:33,412 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-15 02:55:33,412 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-15 02:55:33,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-15 02:55:33,412 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:33,412 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-15 02:55:33,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:55:33,412 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:33,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-15 02:55:33,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:55:33,413 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-15 02:55:33,413 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-15 02:55:33,413 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-15 02:55:33,413 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-15 02:55:33,413 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-15 02:55:33,413 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-15 02:55:33,413 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-15 02:55:33,413 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-15 02:55:33,413 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681527331158.a29fa882a9e3f50c391e3aa08c5c5e72. 2023-04-15 02:55:33,413 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681527331158.a29fa882a9e3f50c391e3aa08c5c5e72. started... 2023-04-15 02:55:33,413 INFO [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing a29fa882a9e3f50c391e3aa08c5c5e72 3/3 column families, dataSize=154.31 KB heapSize=430.69 KB 2023-04-15 02:55:33,413 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a29fa882a9e3f50c391e3aa08c5c5e72, store=A 2023-04-15 02:55:33,414 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-15 02:55:33,414 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a29fa882a9e3f50c391e3aa08c5c5e72, store=B 2023-04-15 02:55:33,414 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-15 02:55:33,414 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a29fa882a9e3f50c391e3aa08c5c5e72, store=C 2023-04-15 02:55:33,414 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-15 02:55:33,556 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34681] regionserver.HRegion(9158): Flush requested on a29fa882a9e3f50c391e3aa08c5c5e72 2023-04-15 02:55:33,557 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681527331158.a29fa882a9e3f50c391e3aa08c5c5e72. as already flushing 2023-04-15 02:55:33,563 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a29fa882a9e3f50c391e3aa08c5c5e72, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:55:33,563 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34681] ipc.CallRunner(144): callId: 18 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:40210 deadline: 1681527393557, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a29fa882a9e3f50c391e3aa08c5c5e72, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:33,563 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a29fa882a9e3f50c391e3aa08c5c5e72, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:55:33,563 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34681] ipc.CallRunner(144): callId: 18 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:40212 deadline: 1681527393560, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a29fa882a9e3f50c391e3aa08c5c5e72, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:33,566 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a29fa882a9e3f50c391e3aa08c5c5e72, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:55:33,566 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34681] ipc.CallRunner(144): callId: 20 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:40192 deadline: 1681527393561, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a29fa882a9e3f50c391e3aa08c5c5e72, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:33,566 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a29fa882a9e3f50c391e3aa08c5c5e72, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:55:33,566 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34681] ipc.CallRunner(144): callId: 18 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:40234 deadline: 1681527393562, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a29fa882a9e3f50c391e3aa08c5c5e72, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:33,567 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a29fa882a9e3f50c391e3aa08c5c5e72, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:55:33,567 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34681] ipc.CallRunner(144): callId: 20 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:40158 deadline: 1681527393562, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a29fa882a9e3f50c391e3aa08c5c5e72, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:33,666 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a29fa882a9e3f50c391e3aa08c5c5e72, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:55:33,666 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34681] ipc.CallRunner(144): callId: 20 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:40210 deadline: 1681527393664, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a29fa882a9e3f50c391e3aa08c5c5e72, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:33,666 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a29fa882a9e3f50c391e3aa08c5c5e72, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:55:33,667 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34681] ipc.CallRunner(144): callId: 20 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:40212 deadline: 1681527393664, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a29fa882a9e3f50c391e3aa08c5c5e72, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:33,670 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a29fa882a9e3f50c391e3aa08c5c5e72, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:55:33,670 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34681] ipc.CallRunner(144): callId: 22 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:40192 deadline: 1681527393667, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a29fa882a9e3f50c391e3aa08c5c5e72, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:33,670 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a29fa882a9e3f50c391e3aa08c5c5e72, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:55:33,670 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34681] ipc.CallRunner(144): callId: 20 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:40234 deadline: 1681527393667, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a29fa882a9e3f50c391e3aa08c5c5e72, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:33,671 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a29fa882a9e3f50c391e3aa08c5c5e72, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:55:33,671 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34681] ipc.CallRunner(144): callId: 22 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:40158 deadline: 1681527393668, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a29fa882a9e3f50c391e3aa08c5c5e72, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:33,820 INFO [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=51.44 KB at sequenceid=46 (bloomFilter=true), to=hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/a29fa882a9e3f50c391e3aa08c5c5e72/.tmp/A/8b1cd7b07fb14077863a5c54dada63bb 2023-04-15 02:55:33,870 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a29fa882a9e3f50c391e3aa08c5c5e72, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:55:33,870 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34681] ipc.CallRunner(144): callId: 22 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:40212 deadline: 1681527393868, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a29fa882a9e3f50c391e3aa08c5c5e72, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:33,872 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a29fa882a9e3f50c391e3aa08c5c5e72, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:55:33,872 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34681] ipc.CallRunner(144): callId: 22 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:40210 deadline: 1681527393868, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a29fa882a9e3f50c391e3aa08c5c5e72, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:33,876 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a29fa882a9e3f50c391e3aa08c5c5e72, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:55:33,877 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34681] ipc.CallRunner(144): callId: 22 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:40234 deadline: 1681527393871, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a29fa882a9e3f50c391e3aa08c5c5e72, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:33,877 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a29fa882a9e3f50c391e3aa08c5c5e72, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:55:33,877 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34681] ipc.CallRunner(144): callId: 24 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:40158 deadline: 1681527393872, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a29fa882a9e3f50c391e3aa08c5c5e72, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:33,877 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a29fa882a9e3f50c391e3aa08c5c5e72, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:55:33,877 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34681] ipc.CallRunner(144): callId: 24 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:40192 deadline: 1681527393872, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a29fa882a9e3f50c391e3aa08c5c5e72, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:34,174 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a29fa882a9e3f50c391e3aa08c5c5e72, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:55:34,174 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34681] ipc.CallRunner(144): callId: 24 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:40212 deadline: 1681527394172, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a29fa882a9e3f50c391e3aa08c5c5e72, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:34,178 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a29fa882a9e3f50c391e3aa08c5c5e72, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:55:34,179 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34681] ipc.CallRunner(144): callId: 24 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:40210 deadline: 1681527394175, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a29fa882a9e3f50c391e3aa08c5c5e72, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:34,182 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a29fa882a9e3f50c391e3aa08c5c5e72, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:55:34,182 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34681] ipc.CallRunner(144): callId: 24 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:40234 deadline: 1681527394179, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a29fa882a9e3f50c391e3aa08c5c5e72, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:34,184 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a29fa882a9e3f50c391e3aa08c5c5e72, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:55:34,184 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34681] ipc.CallRunner(144): callId: 26 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:40158 deadline: 1681527394180, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a29fa882a9e3f50c391e3aa08c5c5e72, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:34,184 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a29fa882a9e3f50c391e3aa08c5c5e72, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:55:34,184 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34681] ipc.CallRunner(144): callId: 26 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:40192 deadline: 1681527394180, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a29fa882a9e3f50c391e3aa08c5c5e72, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:34,229 INFO [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=51.44 KB at sequenceid=46 (bloomFilter=true), to=hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/a29fa882a9e3f50c391e3aa08c5c5e72/.tmp/B/d17591d53587463eb818d66612ab4d0b 2023-04-15 02:55:34,638 INFO [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=51.44 KB at sequenceid=46 (bloomFilter=true), to=hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/a29fa882a9e3f50c391e3aa08c5c5e72/.tmp/C/0a83cb0c7e0f46f6932c2f706cf22c5f 2023-04-15 02:55:34,641 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/a29fa882a9e3f50c391e3aa08c5c5e72/.tmp/A/8b1cd7b07fb14077863a5c54dada63bb as hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/a29fa882a9e3f50c391e3aa08c5c5e72/A/8b1cd7b07fb14077863a5c54dada63bb 2023-04-15 02:55:34,644 INFO [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/a29fa882a9e3f50c391e3aa08c5c5e72/A/8b1cd7b07fb14077863a5c54dada63bb, entries=150, sequenceid=46, filesize=11.6 K 2023-04-15 02:55:34,644 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/a29fa882a9e3f50c391e3aa08c5c5e72/.tmp/B/d17591d53587463eb818d66612ab4d0b as hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/a29fa882a9e3f50c391e3aa08c5c5e72/B/d17591d53587463eb818d66612ab4d0b 2023-04-15 02:55:34,647 INFO [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/a29fa882a9e3f50c391e3aa08c5c5e72/B/d17591d53587463eb818d66612ab4d0b, entries=150, sequenceid=46, filesize=11.6 K 2023-04-15 02:55:34,648 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/a29fa882a9e3f50c391e3aa08c5c5e72/.tmp/C/0a83cb0c7e0f46f6932c2f706cf22c5f as hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/a29fa882a9e3f50c391e3aa08c5c5e72/C/0a83cb0c7e0f46f6932c2f706cf22c5f 2023-04-15 02:55:34,650 INFO [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/a29fa882a9e3f50c391e3aa08c5c5e72/C/0a83cb0c7e0f46f6932c2f706cf22c5f, entries=150, sequenceid=46, filesize=11.6 K 2023-04-15 02:55:34,651 INFO [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~154.31 KB/158010, heapSize ~430.64 KB/440976, currentSize=33.54 KB/34350 for a29fa882a9e3f50c391e3aa08c5c5e72 in 1238ms, sequenceid=46, compaction requested=false 2023-04-15 02:55:34,651 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for a29fa882a9e3f50c391e3aa08c5c5e72: 2023-04-15 02:55:34,651 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681527331158.a29fa882a9e3f50c391e3aa08c5c5e72. 2023-04-15 02:55:34,651 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-15 02:55:34,651 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-15 02:55:34,651 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:34,651 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-15 02:55:34,651 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34681,1681527209133' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-15 02:55:34,653 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:34,653 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:34,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:34,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:55:34,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:55:34,653 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:34,653 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-15 02:55:34,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:55:34,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:55:34,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:55:34,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:34,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:55:34,654 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34681,1681527209133' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-15 02:55:34,654 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@35b084bd[Count = 0] remaining members to acquire global barrier 2023-04-15 02:55:34,654 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-15 02:55:34,655 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:34,655 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:34,655 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:34,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:34,655 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-15 02:55:34,655 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-15 02:55:34,655 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34681,1681527209133' in zk 2023-04-15 02:55:34,655 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:34,656 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-15 02:55:34,656 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-15 02:55:34,656 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-15 02:55:34,656 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:34,656 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-15 02:55:34,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:34,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:55:34,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:55:34,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:55:34,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:55:34,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:55:34,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:34,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:55:34,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:55:34,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:34,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34681,1681527209133': 2023-04-15 02:55:34,658 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34681,1681527209133' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-15 02:55:34,658 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-15 02:55:34,658 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-15 02:55:34,658 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-15 02:55:34,658 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-15 02:55:34,658 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-15 02:55:34,659 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:34,659 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-15 02:55:34,659 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:34,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:34,659 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:34,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:34,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:55:34,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:55:34,660 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-15 02:55:34,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-15 02:55:34,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:55:34,660 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:34,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:34,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:55:34,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:55:34,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:55:34,661 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:34,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:34,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:55:34,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:55:34,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:34,665 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-15 02:55:34,665 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-15 02:55:34,665 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-15 02:55:34,665 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:34,665 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-15 02:55:34,665 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-15 02:55:34,665 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-15 02:55:34,665 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:55:34,665 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:55:34,665 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:34,665 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:34,665 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:34,665 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:34,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-15 02:55:34,665 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-15 02:55:34,665 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-15 02:55:34,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-15 02:55:34,666 DEBUG [Thread-2739] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-15 02:55:34,666 DEBUG [Thread-2739] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-15 02:55:34,683 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34681] regionserver.HRegion(9158): Flush requested on a29fa882a9e3f50c391e3aa08c5c5e72 2023-04-15 02:55:34,683 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing a29fa882a9e3f50c391e3aa08c5c5e72 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-15 02:55:34,684 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a29fa882a9e3f50c391e3aa08c5c5e72, store=A 2023-04-15 02:55:34,684 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-15 02:55:34,684 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a29fa882a9e3f50c391e3aa08c5c5e72, store=B 2023-04-15 02:55:34,684 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-15 02:55:34,684 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a29fa882a9e3f50c391e3aa08c5c5e72, store=C 2023-04-15 02:55:34,684 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-15 02:55:34,691 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=57 (bloomFilter=true), to=hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/a29fa882a9e3f50c391e3aa08c5c5e72/.tmp/A/4f1a4b7179234562828b589c7d2475e0 2023-04-15 02:55:34,700 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=57 (bloomFilter=true), to=hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/a29fa882a9e3f50c391e3aa08c5c5e72/.tmp/B/043e5d05a3184d83a9424af96a48be54 2023-04-15 02:55:34,715 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1681527334700, max timestamp=1681527334713Num uniques 859; 2023-04-15 02:55:34,715 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-3] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-15 02:55:34,715 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1681527334700, max timestamp=1681527334713Num uniques 859; 2023-04-15 02:55:34,715 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-6] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-15 02:55:34,715 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1681527334700, max timestamp=1681527334713Num uniques 859; 2023-04-15 02:55:34,715 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-4] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-15 02:55:34,740 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a29fa882a9e3f50c391e3aa08c5c5e72, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:55:34,740 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a29fa882a9e3f50c391e3aa08c5c5e72, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:55:34,740 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34681] ipc.CallRunner(144): callId: 33 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:40210 deadline: 1681527394731, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a29fa882a9e3f50c391e3aa08c5c5e72, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:34,740 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34681] ipc.CallRunner(144): callId: 34 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:40158 deadline: 1681527394731, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a29fa882a9e3f50c391e3aa08c5c5e72, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:34,740 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a29fa882a9e3f50c391e3aa08c5c5e72, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:55:34,740 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34681] ipc.CallRunner(144): callId: 34 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:40192 deadline: 1681527394734, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a29fa882a9e3f50c391e3aa08c5c5e72, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:34,744 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a29fa882a9e3f50c391e3aa08c5c5e72, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:55:34,744 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34681] ipc.CallRunner(144): callId: 35 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:40212 deadline: 1681527394738, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a29fa882a9e3f50c391e3aa08c5c5e72, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:34,745 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a29fa882a9e3f50c391e3aa08c5c5e72, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:55:34,745 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34681] ipc.CallRunner(144): callId: 33 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:40234 deadline: 1681527394738, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a29fa882a9e3f50c391e3aa08c5c5e72, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:34,766 DEBUG [Thread-2739] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-15 02:55:34,766 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-15 02:55:34,768 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-15 02:55:34,769 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-15 02:55:34,769 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-15 02:55:34,769 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-15 02:55:34,769 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-15 02:55:34,769 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-15 02:55:34,770 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-15 02:55:34,778 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:34,778 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:55:34,779 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-15 02:55:34,779 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:34,779 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-15 02:55:34,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-15 02:55:34,779 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:34,779 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-15 02:55:34,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:55:34,779 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:34,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-15 02:55:34,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:55:34,780 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-15 02:55:34,781 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-15 02:55:34,781 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-15 02:55:34,781 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-15 02:55:34,781 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-15 02:55:34,781 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-15 02:55:34,781 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-15 02:55:34,781 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-15 02:55:34,781 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681527331158.a29fa882a9e3f50c391e3aa08c5c5e72. 2023-04-15 02:55:34,781 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681527331158.a29fa882a9e3f50c391e3aa08c5c5e72. started... 2023-04-15 02:55:34,781 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681527331158.a29fa882a9e3f50c391e3aa08c5c5e72. as already flushing 2023-04-15 02:55:34,781 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681527331158.a29fa882a9e3f50c391e3aa08c5c5e72. 2023-04-15 02:55:34,782 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-15 02:55:34,782 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-15 02:55:34,782 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:34,782 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-15 02:55:34,782 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34681,1681527209133' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-15 02:55:34,782 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:34,782 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:34,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:34,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:55:34,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:55:34,783 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:34,783 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-15 02:55:34,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:55:34,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:55:34,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:55:34,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:34,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:55:34,784 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34681,1681527209133' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-15 02:55:34,784 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-15 02:55:34,784 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@12387980[Count = 0] remaining members to acquire global barrier 2023-04-15 02:55:34,784 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:34,784 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:34,784 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:34,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:34,784 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:34,784 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-15 02:55:34,784 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-15 02:55:34,784 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-15 02:55:34,784 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34681,1681527209133' in zk 2023-04-15 02:55:34,785 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:34,785 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-15 02:55:34,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:34,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:55:34,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:55:34,785 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-15 02:55:34,785 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-15 02:55:34,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:55:34,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:55:34,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:55:34,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:34,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:55:34,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:55:34,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:34,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34681,1681527209133': 2023-04-15 02:55:34,787 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34681,1681527209133' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-15 02:55:34,787 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-15 02:55:34,787 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-15 02:55:34,787 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-15 02:55:34,787 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-15 02:55:34,787 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-15 02:55:34,788 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:34,788 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:34,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:34,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:55:34,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:55:34,788 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:34,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:34,788 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-15 02:55:34,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:55:34,788 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-15 02:55:34,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-15 02:55:34,788 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:34,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:55:34,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:34,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:55:34,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:55:34,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:34,789 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:34,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:55:34,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:55:34,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:34,791 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:34,791 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-15 02:55:34,791 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:55:34,791 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-15 02:55:34,791 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-15 02:55:34,791 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-15 02:55:34,791 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-15 02:55:34,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-15 02:55:34,791 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-15 02:55:34,791 DEBUG [Thread-2739] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-15 02:55:34,791 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:55:34,791 DEBUG [Thread-2739] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-15 02:55:34,791 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-15 02:55:34,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-15 02:55:34,791 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:34,791 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:34,791 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:34,791 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:34,849 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a29fa882a9e3f50c391e3aa08c5c5e72, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:55:34,850 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34681] ipc.CallRunner(144): callId: 36 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:40158 deadline: 1681527394841, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a29fa882a9e3f50c391e3aa08c5c5e72, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:34,850 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a29fa882a9e3f50c391e3aa08c5c5e72, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:55:34,850 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a29fa882a9e3f50c391e3aa08c5c5e72, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:55:34,850 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34681] ipc.CallRunner(144): callId: 36 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:40192 deadline: 1681527394841, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a29fa882a9e3f50c391e3aa08c5c5e72, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:34,850 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34681] ipc.CallRunner(144): callId: 35 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:40210 deadline: 1681527394841, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a29fa882a9e3f50c391e3aa08c5c5e72, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:34,850 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a29fa882a9e3f50c391e3aa08c5c5e72, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:55:34,850 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34681] ipc.CallRunner(144): callId: 37 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:40212 deadline: 1681527394845, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a29fa882a9e3f50c391e3aa08c5c5e72, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:34,850 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a29fa882a9e3f50c391e3aa08c5c5e72, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:55:34,851 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34681] ipc.CallRunner(144): callId: 35 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:40234 deadline: 1681527394845, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a29fa882a9e3f50c391e3aa08c5c5e72, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:34,891 DEBUG [Thread-2739] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-15 02:55:34,892 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-15 02:55:34,893 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-15 02:55:34,894 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-15 02:55:34,894 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-15 02:55:34,894 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-15 02:55:34,894 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-15 02:55:34,894 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-15 02:55:34,894 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-15 02:55:34,894 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:34,894 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:55:34,895 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:34,895 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-15 02:55:34,895 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-15 02:55:34,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-15 02:55:34,895 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:34,895 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-15 02:55:34,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:55:34,895 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:34,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-15 02:55:34,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:55:34,896 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-15 02:55:34,896 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-15 02:55:34,896 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-15 02:55:34,896 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-15 02:55:34,896 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-15 02:55:34,896 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-15 02:55:34,896 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-15 02:55:34,896 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-15 02:55:34,896 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681527331158.a29fa882a9e3f50c391e3aa08c5c5e72. 2023-04-15 02:55:34,896 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681527331158.a29fa882a9e3f50c391e3aa08c5c5e72. started... 2023-04-15 02:55:34,896 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681527331158.a29fa882a9e3f50c391e3aa08c5c5e72. as already flushing 2023-04-15 02:55:34,896 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681527331158.a29fa882a9e3f50c391e3aa08c5c5e72. 2023-04-15 02:55:34,896 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-15 02:55:34,896 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-15 02:55:34,896 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:34,896 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-15 02:55:34,896 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34681,1681527209133' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-15 02:55:34,897 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:34,897 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:34,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:34,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:55:34,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:55:34,897 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:34,897 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-15 02:55:34,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:55:34,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:55:34,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:55:34,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:34,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:55:34,898 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34681,1681527209133' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-15 02:55:34,898 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-15 02:55:34,898 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1aa8185[Count = 0] remaining members to acquire global barrier 2023-04-15 02:55:34,898 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:34,899 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:34,899 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:34,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:34,899 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-15 02:55:34,899 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-15 02:55:34,899 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:34,899 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-15 02:55:34,899 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34681,1681527209133' in zk 2023-04-15 02:55:34,900 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:34,900 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-15 02:55:34,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:34,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:55:34,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:55:34,900 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-15 02:55:34,900 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-15 02:55:34,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:55:34,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:55:34,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:55:34,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:34,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:55:34,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:55:34,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:34,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34681,1681527209133': 2023-04-15 02:55:34,901 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34681,1681527209133' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-15 02:55:34,901 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-15 02:55:34,901 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-15 02:55:34,901 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-15 02:55:34,901 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-15 02:55:34,901 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-15 02:55:34,902 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:34,902 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:34,902 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:34,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:34,902 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-15 02:55:34,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:34,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:55:34,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:55:34,903 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-15 02:55:34,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-15 02:55:34,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:55:34,903 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:34,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:34,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:55:34,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:55:34,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:55:34,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:34,904 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:34,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:55:34,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:55:34,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:34,905 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-15 02:55:34,905 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:34,905 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-15 02:55:34,906 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-15 02:55:34,906 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-15 02:55:34,905 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-15 02:55:34,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-15 02:55:34,906 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-15 02:55:34,906 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-15 02:55:34,905 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:55:34,906 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:55:34,906 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:34,906 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:34,906 DEBUG [Thread-2739] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-15 02:55:34,906 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-15 02:55:34,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-15 02:55:34,906 DEBUG [Thread-2739] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-15 02:55:34,906 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:34,906 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:35,006 DEBUG [Thread-2739] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-15 02:55:35,007 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-15 02:55:35,008 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-15 02:55:35,009 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-15 02:55:35,009 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-15 02:55:35,009 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-15 02:55:35,009 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-15 02:55:35,009 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-15 02:55:35,009 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-15 02:55:35,009 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:35,009 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:55:35,010 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-15 02:55:35,010 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:35,010 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-15 02:55:35,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-15 02:55:35,010 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:35,010 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-15 02:55:35,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:55:35,010 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:35,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-15 02:55:35,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:55:35,011 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-15 02:55:35,011 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-15 02:55:35,011 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-15 02:55:35,011 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-15 02:55:35,011 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-15 02:55:35,011 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-15 02:55:35,011 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-15 02:55:35,011 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-15 02:55:35,011 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681527331158.a29fa882a9e3f50c391e3aa08c5c5e72. 2023-04-15 02:55:35,011 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681527331158.a29fa882a9e3f50c391e3aa08c5c5e72. started... 2023-04-15 02:55:35,011 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681527331158.a29fa882a9e3f50c391e3aa08c5c5e72. as already flushing 2023-04-15 02:55:35,011 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681527331158.a29fa882a9e3f50c391e3aa08c5c5e72. 2023-04-15 02:55:35,012 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-15 02:55:35,012 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-15 02:55:35,012 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:35,012 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-15 02:55:35,012 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34681,1681527209133' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-15 02:55:35,012 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:35,012 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:35,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:35,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:55:35,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:55:35,013 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:35,013 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-15 02:55:35,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:55:35,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:55:35,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:55:35,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:35,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:55:35,014 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34681,1681527209133' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-15 02:55:35,014 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@623a1824[Count = 0] remaining members to acquire global barrier 2023-04-15 02:55:35,014 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-15 02:55:35,014 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:35,014 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:35,014 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:35,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:35,014 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-15 02:55:35,014 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-15 02:55:35,014 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34681,1681527209133' in zk 2023-04-15 02:55:35,015 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-15 02:55:35,015 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-15 02:55:35,015 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-15 02:55:35,015 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:35,015 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34681,1681527209133' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-15 02:55:35,016 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-15 02:55:35,016 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-15 02:55:35,016 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-15 02:55:35,016 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-15 02:55:35,016 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-15 02:55:35,016 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-15 02:55:35,016 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:35,016 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:35,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:35,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:55:35,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:55:35,016 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:35,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:35,016 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-15 02:55:35,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:55:35,017 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-15 02:55:35,017 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:35,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:55:35,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-15 02:55:35,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:55:35,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:35,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:55:35,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:35,018 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:35,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:55:35,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:55:35,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:35,019 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:35,019 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:55:35,019 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:55:35,019 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-15 02:55:35,019 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-15 02:55:35,019 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:35,019 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-15 02:55:35,020 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-15 02:55:35,020 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:35,020 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-15 02:55:35,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-15 02:55:35,019 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-15 02:55:35,020 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:35,020 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:35,020 DEBUG [Thread-2739] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-15 02:55:35,020 DEBUG [Thread-2739] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-15 02:55:35,020 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-15 02:55:35,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-15 02:55:35,055 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a29fa882a9e3f50c391e3aa08c5c5e72, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:55:35,055 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34681] ipc.CallRunner(144): callId: 37 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:40210 deadline: 1681527395051, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a29fa882a9e3f50c391e3aa08c5c5e72, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:35,055 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a29fa882a9e3f50c391e3aa08c5c5e72, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:55:35,055 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34681] ipc.CallRunner(144): callId: 38 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:40158 deadline: 1681527395052, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a29fa882a9e3f50c391e3aa08c5c5e72, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:35,055 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a29fa882a9e3f50c391e3aa08c5c5e72, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:55:35,056 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34681] ipc.CallRunner(144): callId: 38 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:40192 deadline: 1681527395052, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a29fa882a9e3f50c391e3aa08c5c5e72, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:35,056 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a29fa882a9e3f50c391e3aa08c5c5e72, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:55:35,056 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34681] ipc.CallRunner(144): callId: 39 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:40212 deadline: 1681527395052, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a29fa882a9e3f50c391e3aa08c5c5e72, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:35,056 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a29fa882a9e3f50c391e3aa08c5c5e72, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:55:35,056 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34681] ipc.CallRunner(144): callId: 37 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:40234 deadline: 1681527395052, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a29fa882a9e3f50c391e3aa08c5c5e72, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:35,120 DEBUG [Thread-2739] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-15 02:55:35,120 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-15 02:55:35,121 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-15 02:55:35,122 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-15 02:55:35,123 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-15 02:55:35,123 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-15 02:55:35,123 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-15 02:55:35,123 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-15 02:55:35,123 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-15 02:55:35,123 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:35,123 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:55:35,124 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=57 (bloomFilter=true), to=hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/a29fa882a9e3f50c391e3aa08c5c5e72/.tmp/C/a9f1e71f1b484838afc01f678f7f2456 2023-04-15 02:55:35,124 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:35,124 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-15 02:55:35,124 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-15 02:55:35,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-15 02:55:35,125 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:35,125 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-15 02:55:35,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:55:35,125 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:35,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-15 02:55:35,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:55:35,126 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-15 02:55:35,126 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-15 02:55:35,126 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-15 02:55:35,126 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-15 02:55:35,126 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-15 02:55:35,126 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-15 02:55:35,127 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-15 02:55:35,127 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-15 02:55:35,127 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681527331158.a29fa882a9e3f50c391e3aa08c5c5e72. 2023-04-15 02:55:35,127 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681527331158.a29fa882a9e3f50c391e3aa08c5c5e72. started... 2023-04-15 02:55:35,127 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681527331158.a29fa882a9e3f50c391e3aa08c5c5e72. as already flushing 2023-04-15 02:55:35,127 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681527331158.a29fa882a9e3f50c391e3aa08c5c5e72. 2023-04-15 02:55:35,127 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-15 02:55:35,127 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-15 02:55:35,127 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/a29fa882a9e3f50c391e3aa08c5c5e72/.tmp/A/4f1a4b7179234562828b589c7d2475e0 as hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/a29fa882a9e3f50c391e3aa08c5c5e72/A/4f1a4b7179234562828b589c7d2475e0 2023-04-15 02:55:35,128 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:35,128 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-15 02:55:35,128 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34681,1681527209133' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-15 02:55:35,128 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:35,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:35,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:55:35,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:55:35,128 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:35,129 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:35,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:55:35,129 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-15 02:55:35,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:55:35,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:55:35,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:35,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:55:35,130 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34681,1681527209133' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-15 02:55:35,130 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@531efdb2[Count = 0] remaining members to acquire global barrier 2023-04-15 02:55:35,130 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-15 02:55:35,130 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:35,130 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:35,130 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:35,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:35,131 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-15 02:55:35,131 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-15 02:55:35,131 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/a29fa882a9e3f50c391e3aa08c5c5e72/A/4f1a4b7179234562828b589c7d2475e0, entries=200, sequenceid=57, filesize=13.9 K 2023-04-15 02:55:35,131 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:35,131 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-15 02:55:35,131 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34681,1681527209133' in zk 2023-04-15 02:55:35,131 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/a29fa882a9e3f50c391e3aa08c5c5e72/.tmp/B/043e5d05a3184d83a9424af96a48be54 as hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/a29fa882a9e3f50c391e3aa08c5c5e72/B/043e5d05a3184d83a9424af96a48be54 2023-04-15 02:55:35,131 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:35,132 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-15 02:55:35,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:35,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:55:35,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:55:35,132 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-15 02:55:35,132 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-15 02:55:35,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:55:35,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:55:35,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:55:35,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:35,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:55:35,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:55:35,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:35,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34681,1681527209133': 2023-04-15 02:55:35,134 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34681,1681527209133' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-15 02:55:35,134 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-15 02:55:35,134 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-15 02:55:35,134 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-15 02:55:35,134 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-15 02:55:35,134 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-15 02:55:35,135 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:35,135 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-15 02:55:35,135 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:35,135 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:35,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:35,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:35,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:55:35,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:55:35,135 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/a29fa882a9e3f50c391e3aa08c5c5e72/B/043e5d05a3184d83a9424af96a48be54, entries=150, sequenceid=57, filesize=11.6 K 2023-04-15 02:55:35,135 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-15 02:55:35,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-15 02:55:35,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:55:35,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:35,136 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:35,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:55:35,136 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/a29fa882a9e3f50c391e3aa08c5c5e72/.tmp/C/a9f1e71f1b484838afc01f678f7f2456 as hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/a29fa882a9e3f50c391e3aa08c5c5e72/C/a9f1e71f1b484838afc01f678f7f2456 2023-04-15 02:55:35,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:55:35,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:55:35,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:35,136 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:35,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:55:35,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:55:35,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:35,138 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:35,138 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:55:35,138 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:55:35,138 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-15 02:55:35,138 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:35,138 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:35,138 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-15 02:55:35,138 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-15 02:55:35,138 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-15 02:55:35,138 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-15 02:55:35,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-15 02:55:35,138 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-15 02:55:35,138 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:35,138 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:35,138 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-15 02:55:35,138 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-15 02:55:35,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-15 02:55:35,138 DEBUG [Thread-2739] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-15 02:55:35,139 DEBUG [Thread-2739] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-15 02:55:35,139 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/a29fa882a9e3f50c391e3aa08c5c5e72/C/a9f1e71f1b484838afc01f678f7f2456, entries=150, sequenceid=57, filesize=11.6 K 2023-04-15 02:55:35,140 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=127.47 KB/130530 for a29fa882a9e3f50c391e3aa08c5c5e72 in 457ms, sequenceid=57, compaction requested=true 2023-04-15 02:55:35,140 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for a29fa882a9e3f50c391e3aa08c5c5e72: 2023-04-15 02:55:35,141 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-15 02:55:35,141 DEBUG [RS:0;jenkins-hbase20:34681-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-15 02:55:35,141 DEBUG [RS:0;jenkins-hbase20:34681-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-15 02:55:35,141 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-15 02:55:35,141 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-15 02:55:35,142 DEBUG [RS:0;jenkins-hbase20:34681-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 35658 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-15 02:55:35,142 DEBUG [RS:0;jenkins-hbase20:34681-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37998 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-15 02:55:35,142 DEBUG [RS:0;jenkins-hbase20:34681-longCompactions-0] regionserver.HStore(1912): a29fa882a9e3f50c391e3aa08c5c5e72/B is initiating minor compaction (all files) 2023-04-15 02:55:35,142 DEBUG [RS:0;jenkins-hbase20:34681-shortCompactions-0] regionserver.HStore(1912): a29fa882a9e3f50c391e3aa08c5c5e72/A is initiating minor compaction (all files) 2023-04-15 02:55:35,142 INFO [RS:0;jenkins-hbase20:34681-longCompactions-0] regionserver.HRegion(2259): Starting compaction of a29fa882a9e3f50c391e3aa08c5c5e72/B in TestAcidGuarantees,,1681527331158.a29fa882a9e3f50c391e3aa08c5c5e72. 2023-04-15 02:55:35,142 INFO [RS:0;jenkins-hbase20:34681-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of a29fa882a9e3f50c391e3aa08c5c5e72/A in TestAcidGuarantees,,1681527331158.a29fa882a9e3f50c391e3aa08c5c5e72. 2023-04-15 02:55:35,142 INFO [RS:0;jenkins-hbase20:34681-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/a29fa882a9e3f50c391e3aa08c5c5e72/B/7c8b6c7189e14c99b6df874cccccf3e4, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/a29fa882a9e3f50c391e3aa08c5c5e72/B/d17591d53587463eb818d66612ab4d0b, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/a29fa882a9e3f50c391e3aa08c5c5e72/B/043e5d05a3184d83a9424af96a48be54] into tmpdir=hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/a29fa882a9e3f50c391e3aa08c5c5e72/.tmp, totalSize=34.8 K 2023-04-15 02:55:35,142 INFO [RS:0;jenkins-hbase20:34681-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/a29fa882a9e3f50c391e3aa08c5c5e72/A/0e96c61d77dc4e7196e63b48bf12904c, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/a29fa882a9e3f50c391e3aa08c5c5e72/A/8b1cd7b07fb14077863a5c54dada63bb, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/a29fa882a9e3f50c391e3aa08c5c5e72/A/4f1a4b7179234562828b589c7d2475e0] into tmpdir=hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/a29fa882a9e3f50c391e3aa08c5c5e72/.tmp, totalSize=37.1 K 2023-04-15 02:55:35,142 DEBUG [RS:0;jenkins-hbase20:34681-longCompactions-0] compactions.Compactor(207): Compacting 7c8b6c7189e14c99b6df874cccccf3e4, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=9, earliestPutTs=1681527332386 2023-04-15 02:55:35,142 DEBUG [RS:0;jenkins-hbase20:34681-shortCompactions-0] compactions.Compactor(207): Compacting 0e96c61d77dc4e7196e63b48bf12904c, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=9, earliestPutTs=1681527332386 2023-04-15 02:55:35,142 DEBUG [RS:0;jenkins-hbase20:34681-longCompactions-0] compactions.Compactor(207): Compacting d17591d53587463eb818d66612ab4d0b, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=46, earliestPutTs=1681527332432 2023-04-15 02:55:35,142 DEBUG [RS:0;jenkins-hbase20:34681-shortCompactions-0] compactions.Compactor(207): Compacting 8b1cd7b07fb14077863a5c54dada63bb, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=46, earliestPutTs=1681527332432 2023-04-15 02:55:35,143 DEBUG [RS:0;jenkins-hbase20:34681-longCompactions-0] compactions.Compactor(207): Compacting 043e5d05a3184d83a9424af96a48be54, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=57, earliestPutTs=1681527334679 2023-04-15 02:55:35,143 DEBUG [RS:0;jenkins-hbase20:34681-shortCompactions-0] compactions.Compactor(207): Compacting 4f1a4b7179234562828b589c7d2475e0, keycount=200, bloomtype=ROW, size=13.9 K, encoding=NONE, compression=NONE, seqNum=57, earliestPutTs=1681527333559 2023-04-15 02:55:35,148 INFO [RS:0;jenkins-hbase20:34681-longCompactions-0] throttle.PressureAwareThroughputController(145): a29fa882a9e3f50c391e3aa08c5c5e72#B#compaction#353 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-15 02:55:35,149 INFO [RS:0;jenkins-hbase20:34681-shortCompactions-0] throttle.PressureAwareThroughputController(145): a29fa882a9e3f50c391e3aa08c5c5e72#A#compaction#354 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-15 02:55:35,156 DEBUG [RS:0;jenkins-hbase20:34681-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/a29fa882a9e3f50c391e3aa08c5c5e72/.tmp/A/ade7f87389884e4ea915a78f74f55f27 as hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/a29fa882a9e3f50c391e3aa08c5c5e72/A/ade7f87389884e4ea915a78f74f55f27 2023-04-15 02:55:35,159 INFO [RS:0;jenkins-hbase20:34681-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in a29fa882a9e3f50c391e3aa08c5c5e72/A of a29fa882a9e3f50c391e3aa08c5c5e72 into ade7f87389884e4ea915a78f74f55f27(size=11.7 K), total size for store is 11.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-15 02:55:35,159 DEBUG [RS:0;jenkins-hbase20:34681-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for a29fa882a9e3f50c391e3aa08c5c5e72: 2023-04-15 02:55:35,159 INFO [RS:0;jenkins-hbase20:34681-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681527331158.a29fa882a9e3f50c391e3aa08c5c5e72., storeName=a29fa882a9e3f50c391e3aa08c5c5e72/A, priority=13, startTime=1681527335141; duration=0sec 2023-04-15 02:55:35,159 DEBUG [RS:0;jenkins-hbase20:34681-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-15 02:55:35,159 DEBUG [RS:0;jenkins-hbase20:34681-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-15 02:55:35,160 DEBUG [RS:0;jenkins-hbase20:34681-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 35658 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-15 02:55:35,160 DEBUG [RS:0;jenkins-hbase20:34681-shortCompactions-0] regionserver.HStore(1912): a29fa882a9e3f50c391e3aa08c5c5e72/C is initiating minor compaction (all files) 2023-04-15 02:55:35,160 INFO [RS:0;jenkins-hbase20:34681-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of a29fa882a9e3f50c391e3aa08c5c5e72/C in TestAcidGuarantees,,1681527331158.a29fa882a9e3f50c391e3aa08c5c5e72. 2023-04-15 02:55:35,160 INFO [RS:0;jenkins-hbase20:34681-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/a29fa882a9e3f50c391e3aa08c5c5e72/C/4b027f62b2504671b04f87e52c4b6d59, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/a29fa882a9e3f50c391e3aa08c5c5e72/C/0a83cb0c7e0f46f6932c2f706cf22c5f, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/a29fa882a9e3f50c391e3aa08c5c5e72/C/a9f1e71f1b484838afc01f678f7f2456] into tmpdir=hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/a29fa882a9e3f50c391e3aa08c5c5e72/.tmp, totalSize=34.8 K 2023-04-15 02:55:35,160 DEBUG [RS:0;jenkins-hbase20:34681-shortCompactions-0] compactions.Compactor(207): Compacting 4b027f62b2504671b04f87e52c4b6d59, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=9, earliestPutTs=1681527332386 2023-04-15 02:55:35,161 DEBUG [RS:0;jenkins-hbase20:34681-shortCompactions-0] compactions.Compactor(207): Compacting 0a83cb0c7e0f46f6932c2f706cf22c5f, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=46, earliestPutTs=1681527332432 2023-04-15 02:55:35,161 DEBUG [RS:0;jenkins-hbase20:34681-shortCompactions-0] compactions.Compactor(207): Compacting a9f1e71f1b484838afc01f678f7f2456, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=57, earliestPutTs=1681527334679 2023-04-15 02:55:35,165 INFO [RS:0;jenkins-hbase20:34681-shortCompactions-0] throttle.PressureAwareThroughputController(145): a29fa882a9e3f50c391e3aa08c5c5e72#C#compaction#355 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-15 02:55:35,239 DEBUG [Thread-2739] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-15 02:55:35,239 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-15 02:55:35,240 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-15 02:55:35,241 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-15 02:55:35,242 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-15 02:55:35,242 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-15 02:55:35,242 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-15 02:55:35,242 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-15 02:55:35,242 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-15 02:55:35,242 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:35,242 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:55:35,243 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-15 02:55:35,243 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:35,243 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-15 02:55:35,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-15 02:55:35,243 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:35,244 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-15 02:55:35,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:55:35,244 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:35,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-15 02:55:35,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:55:35,244 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-15 02:55:35,244 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-15 02:55:35,244 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-15 02:55:35,244 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-15 02:55:35,244 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-15 02:55:35,244 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-15 02:55:35,245 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-15 02:55:35,245 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-15 02:55:35,245 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681527331158.a29fa882a9e3f50c391e3aa08c5c5e72. 2023-04-15 02:55:35,245 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681527331158.a29fa882a9e3f50c391e3aa08c5c5e72. started... 2023-04-15 02:55:35,245 INFO [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing a29fa882a9e3f50c391e3aa08c5c5e72 3/3 column families, dataSize=127.47 KB heapSize=360.38 KB 2023-04-15 02:55:35,245 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a29fa882a9e3f50c391e3aa08c5c5e72, store=A 2023-04-15 02:55:35,245 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-15 02:55:35,245 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a29fa882a9e3f50c391e3aa08c5c5e72, store=B 2023-04-15 02:55:35,245 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-15 02:55:35,245 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a29fa882a9e3f50c391e3aa08c5c5e72, store=C 2023-04-15 02:55:35,245 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-15 02:55:35,322 WARN [HBase-Metrics2-1] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties 2023-04-15 02:55:35,359 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34681] regionserver.HRegion(9158): Flush requested on a29fa882a9e3f50c391e3aa08c5c5e72 2023-04-15 02:55:35,359 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681527331158.a29fa882a9e3f50c391e3aa08c5c5e72. as already flushing 2023-04-15 02:55:35,368 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a29fa882a9e3f50c391e3aa08c5c5e72, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:55:35,368 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a29fa882a9e3f50c391e3aa08c5c5e72, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:55:35,368 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34681] ipc.CallRunner(144): callId: 40 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:40210 deadline: 1681527395362, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a29fa882a9e3f50c391e3aa08c5c5e72, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:35,368 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34681] ipc.CallRunner(144): callId: 40 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:40234 deadline: 1681527395363, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a29fa882a9e3f50c391e3aa08c5c5e72, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:35,368 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a29fa882a9e3f50c391e3aa08c5c5e72, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:55:35,368 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34681] ipc.CallRunner(144): callId: 42 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:40158 deadline: 1681527395365, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a29fa882a9e3f50c391e3aa08c5c5e72, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:35,372 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a29fa882a9e3f50c391e3aa08c5c5e72, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:55:35,372 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34681] ipc.CallRunner(144): callId: 42 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:40192 deadline: 1681527395366, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a29fa882a9e3f50c391e3aa08c5c5e72, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:35,372 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a29fa882a9e3f50c391e3aa08c5c5e72, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:55:35,372 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34681] ipc.CallRunner(144): callId: 43 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:40212 deadline: 1681527395366, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a29fa882a9e3f50c391e3aa08c5c5e72, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:35,471 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a29fa882a9e3f50c391e3aa08c5c5e72, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:55:35,471 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34681] ipc.CallRunner(144): callId: 42 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:40210 deadline: 1681527395469, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a29fa882a9e3f50c391e3aa08c5c5e72, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:35,471 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a29fa882a9e3f50c391e3aa08c5c5e72, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:55:35,471 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34681] ipc.CallRunner(144): callId: 42 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:40234 deadline: 1681527395469, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a29fa882a9e3f50c391e3aa08c5c5e72, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:35,471 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a29fa882a9e3f50c391e3aa08c5c5e72, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:55:35,472 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34681] ipc.CallRunner(144): callId: 44 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:40158 deadline: 1681527395469, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a29fa882a9e3f50c391e3aa08c5c5e72, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:35,477 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a29fa882a9e3f50c391e3aa08c5c5e72, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:55:35,478 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34681] ipc.CallRunner(144): callId: 44 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:40192 deadline: 1681527395473, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a29fa882a9e3f50c391e3aa08c5c5e72, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:35,478 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a29fa882a9e3f50c391e3aa08c5c5e72, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:55:35,478 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34681] ipc.CallRunner(144): callId: 45 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:40212 deadline: 1681527395473, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a29fa882a9e3f50c391e3aa08c5c5e72, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:35,555 DEBUG [RS:0;jenkins-hbase20:34681-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/a29fa882a9e3f50c391e3aa08c5c5e72/.tmp/B/e3ff06b400854006852039779a95aade as hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/a29fa882a9e3f50c391e3aa08c5c5e72/B/e3ff06b400854006852039779a95aade 2023-04-15 02:55:35,559 INFO [RS:0;jenkins-hbase20:34681-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in a29fa882a9e3f50c391e3aa08c5c5e72/B of a29fa882a9e3f50c391e3aa08c5c5e72 into e3ff06b400854006852039779a95aade(size=11.7 K), total size for store is 11.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-15 02:55:35,559 DEBUG [RS:0;jenkins-hbase20:34681-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for a29fa882a9e3f50c391e3aa08c5c5e72: 2023-04-15 02:55:35,559 INFO [RS:0;jenkins-hbase20:34681-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681527331158.a29fa882a9e3f50c391e3aa08c5c5e72., storeName=a29fa882a9e3f50c391e3aa08c5c5e72/B, priority=13, startTime=1681527335141; duration=0sec 2023-04-15 02:55:35,559 DEBUG [RS:0;jenkins-hbase20:34681-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-15 02:55:35,572 DEBUG [RS:0;jenkins-hbase20:34681-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/a29fa882a9e3f50c391e3aa08c5c5e72/.tmp/C/451fc6fe095f4e2f8ac779de5c121c27 as hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/a29fa882a9e3f50c391e3aa08c5c5e72/C/451fc6fe095f4e2f8ac779de5c121c27 2023-04-15 02:55:35,575 INFO [RS:0;jenkins-hbase20:34681-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in a29fa882a9e3f50c391e3aa08c5c5e72/C of a29fa882a9e3f50c391e3aa08c5c5e72 into 451fc6fe095f4e2f8ac779de5c121c27(size=11.7 K), total size for store is 11.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-15 02:55:35,575 DEBUG [RS:0;jenkins-hbase20:34681-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for a29fa882a9e3f50c391e3aa08c5c5e72: 2023-04-15 02:55:35,575 INFO [RS:0;jenkins-hbase20:34681-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681527331158.a29fa882a9e3f50c391e3aa08c5c5e72., storeName=a29fa882a9e3f50c391e3aa08c5c5e72/C, priority=13, startTime=1681527335141; duration=0sec 2023-04-15 02:55:35,575 DEBUG [RS:0;jenkins-hbase20:34681-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-15 02:55:35,652 INFO [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=93 (bloomFilter=true), to=hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/a29fa882a9e3f50c391e3aa08c5c5e72/.tmp/A/bee8d060091749a4af47caf99098c5bb 2023-04-15 02:55:35,665 INFO [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=93 (bloomFilter=true), to=hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/a29fa882a9e3f50c391e3aa08c5c5e72/.tmp/B/c05c1b5850724aa8b501d9b7acd62a0c 2023-04-15 02:55:35,674 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a29fa882a9e3f50c391e3aa08c5c5e72, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:55:35,675 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34681] ipc.CallRunner(144): callId: 44 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:40210 deadline: 1681527395673, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a29fa882a9e3f50c391e3aa08c5c5e72, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:35,675 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a29fa882a9e3f50c391e3aa08c5c5e72, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:55:35,675 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34681] ipc.CallRunner(144): callId: 44 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:40234 deadline: 1681527395673, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a29fa882a9e3f50c391e3aa08c5c5e72, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:35,675 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a29fa882a9e3f50c391e3aa08c5c5e72, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:55:35,675 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34681] ipc.CallRunner(144): callId: 46 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:40158 deadline: 1681527395673, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a29fa882a9e3f50c391e3aa08c5c5e72, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:35,676 INFO [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=93 (bloomFilter=true), to=hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/a29fa882a9e3f50c391e3aa08c5c5e72/.tmp/C/a55c129a820d464a8dd3cf851be769e5 2023-04-15 02:55:35,680 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/a29fa882a9e3f50c391e3aa08c5c5e72/.tmp/A/bee8d060091749a4af47caf99098c5bb as hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/a29fa882a9e3f50c391e3aa08c5c5e72/A/bee8d060091749a4af47caf99098c5bb 2023-04-15 02:55:35,681 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a29fa882a9e3f50c391e3aa08c5c5e72, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:55:35,681 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34681] ipc.CallRunner(144): callId: 47 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:40212 deadline: 1681527395679, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a29fa882a9e3f50c391e3aa08c5c5e72, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:35,681 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a29fa882a9e3f50c391e3aa08c5c5e72, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:55:35,681 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34681] ipc.CallRunner(144): callId: 46 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:40192 deadline: 1681527395679, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a29fa882a9e3f50c391e3aa08c5c5e72, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:35,683 INFO [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/a29fa882a9e3f50c391e3aa08c5c5e72/A/bee8d060091749a4af47caf99098c5bb, entries=150, sequenceid=93, filesize=11.6 K 2023-04-15 02:55:35,683 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/a29fa882a9e3f50c391e3aa08c5c5e72/.tmp/B/c05c1b5850724aa8b501d9b7acd62a0c as hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/a29fa882a9e3f50c391e3aa08c5c5e72/B/c05c1b5850724aa8b501d9b7acd62a0c 2023-04-15 02:55:35,686 INFO [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/a29fa882a9e3f50c391e3aa08c5c5e72/B/c05c1b5850724aa8b501d9b7acd62a0c, entries=150, sequenceid=93, filesize=11.6 K 2023-04-15 02:55:35,687 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/a29fa882a9e3f50c391e3aa08c5c5e72/.tmp/C/a55c129a820d464a8dd3cf851be769e5 as hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/a29fa882a9e3f50c391e3aa08c5c5e72/C/a55c129a820d464a8dd3cf851be769e5 2023-04-15 02:55:35,692 INFO [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/a29fa882a9e3f50c391e3aa08c5c5e72/C/a55c129a820d464a8dd3cf851be769e5, entries=150, sequenceid=93, filesize=11.6 K 2023-04-15 02:55:35,693 INFO [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~127.47 KB/130530, heapSize ~360.33 KB/368976, currentSize=53.67 KB/54960 for a29fa882a9e3f50c391e3aa08c5c5e72 in 448ms, sequenceid=93, compaction requested=false 2023-04-15 02:55:35,693 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for a29fa882a9e3f50c391e3aa08c5c5e72: 2023-04-15 02:55:35,693 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681527331158.a29fa882a9e3f50c391e3aa08c5c5e72. 2023-04-15 02:55:35,693 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-15 02:55:35,693 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-15 02:55:35,693 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:35,694 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-15 02:55:35,694 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34681,1681527209133' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-15 02:55:35,697 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:35,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:35,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:55:35,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:55:35,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:55:35,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:55:35,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:55:35,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:35,698 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:35,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:55:35,698 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:35,698 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-15 02:55:35,698 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34681,1681527209133' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-15 02:55:35,698 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-15 02:55:35,698 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@23b7a28[Count = 0] remaining members to acquire global barrier 2023-04-15 02:55:35,698 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:35,699 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:35,699 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:35,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:35,699 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-15 02:55:35,699 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-15 02:55:35,699 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34681,1681527209133' in zk 2023-04-15 02:55:35,699 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:35,699 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-15 02:55:35,699 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:35,699 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-15 02:55:35,700 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-15 02:55:35,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:35,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:55:35,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:55:35,700 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-15 02:55:35,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:55:35,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:55:35,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:55:35,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:35,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:55:35,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:55:35,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:35,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34681,1681527209133': 2023-04-15 02:55:35,703 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34681,1681527209133' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-15 02:55:35,703 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-15 02:55:35,703 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-15 02:55:35,703 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-15 02:55:35,703 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-15 02:55:35,703 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-15 02:55:35,704 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:35,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:35,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:55:35,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:55:35,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:55:35,705 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:35,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:55:35,705 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:35,705 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-15 02:55:35,705 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:35,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:35,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:55:35,705 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-15 02:55:35,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-15 02:55:35,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:35,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:55:35,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:35,706 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:35,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:55:35,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:55:35,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:35,708 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:35,708 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:55:35,708 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:55:35,708 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:35,708 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:35,708 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-15 02:55:35,708 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:35,708 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:35,708 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-15 02:55:35,708 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-15 02:55:35,708 DEBUG [Thread-2739] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-15 02:55:35,708 DEBUG [Thread-2739] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-15 02:55:35,708 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-15 02:55:35,708 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-15 02:55:35,708 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-15 02:55:35,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-15 02:55:35,709 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-15 02:55:35,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-15 02:55:35,808 DEBUG [Thread-2739] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-15 02:55:35,809 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-15 02:55:35,810 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-15 02:55:35,812 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-15 02:55:35,812 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-15 02:55:35,812 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-15 02:55:35,812 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-15 02:55:35,812 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-15 02:55:35,812 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-15 02:55:35,812 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:35,812 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:55:35,813 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-15 02:55:35,813 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-15 02:55:35,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-15 02:55:35,813 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:35,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:55:35,813 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:35,813 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-15 02:55:35,814 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:35,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-15 02:55:35,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:55:35,814 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-15 02:55:35,814 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-15 02:55:35,814 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-15 02:55:35,814 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-15 02:55:35,814 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-15 02:55:35,814 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-15 02:55:35,814 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-15 02:55:35,814 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-15 02:55:35,814 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681527331158.a29fa882a9e3f50c391e3aa08c5c5e72. 2023-04-15 02:55:35,814 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681527331158.a29fa882a9e3f50c391e3aa08c5c5e72. started... 2023-04-15 02:55:35,814 INFO [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing a29fa882a9e3f50c391e3aa08c5c5e72 3/3 column families, dataSize=53.67 KB heapSize=160.13 KB 2023-04-15 02:55:35,815 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a29fa882a9e3f50c391e3aa08c5c5e72, store=A 2023-04-15 02:55:35,815 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-15 02:55:35,815 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a29fa882a9e3f50c391e3aa08c5c5e72, store=B 2023-04-15 02:55:35,815 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-15 02:55:35,815 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a29fa882a9e3f50c391e3aa08c5c5e72, store=C 2023-04-15 02:55:35,815 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-15 02:55:35,825 INFO [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=106 (bloomFilter=true), to=hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/a29fa882a9e3f50c391e3aa08c5c5e72/.tmp/A/02ceb979dd0b4fde839b9ab0388624be 2023-04-15 02:55:35,978 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34681] regionserver.HRegion(9158): Flush requested on a29fa882a9e3f50c391e3aa08c5c5e72 2023-04-15 02:55:35,978 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681527331158.a29fa882a9e3f50c391e3aa08c5c5e72. as already flushing 2023-04-15 02:55:36,006 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681527335992, max timestamp=1681527336004Num uniques 859; 2023-04-15 02:55:36,006 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681527335992, max timestamp=1681527336004Num uniques 859; 2023-04-15 02:55:36,006 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-8] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-15 02:55:36,006 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681527335992, max timestamp=1681527336004Num uniques 859; 2023-04-15 02:55:36,006 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-2] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-15 02:55:36,006 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-1] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-15 02:55:36,027 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a29fa882a9e3f50c391e3aa08c5c5e72, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:55:36,027 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34681] ipc.CallRunner(144): callId: 54 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:40192 deadline: 1681527396019, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a29fa882a9e3f50c391e3aa08c5c5e72, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:36,032 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a29fa882a9e3f50c391e3aa08c5c5e72, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:55:36,032 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34681] ipc.CallRunner(144): callId: 52 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:40234 deadline: 1681527396024, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a29fa882a9e3f50c391e3aa08c5c5e72, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:36,032 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a29fa882a9e3f50c391e3aa08c5c5e72, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:55:36,032 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34681] ipc.CallRunner(144): callId: 53 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:40210 deadline: 1681527396025, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a29fa882a9e3f50c391e3aa08c5c5e72, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:36,036 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a29fa882a9e3f50c391e3aa08c5c5e72, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:55:36,036 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34681] ipc.CallRunner(144): callId: 55 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:40158 deadline: 1681527396026, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a29fa882a9e3f50c391e3aa08c5c5e72, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:36,036 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a29fa882a9e3f50c391e3aa08c5c5e72, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:55:36,036 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34681] ipc.CallRunner(144): callId: 56 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:40212 deadline: 1681527396026, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a29fa882a9e3f50c391e3aa08c5c5e72, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:36,128 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a29fa882a9e3f50c391e3aa08c5c5e72, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:55:36,128 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34681] ipc.CallRunner(144): callId: 56 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:40192 deadline: 1681527396128, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a29fa882a9e3f50c391e3aa08c5c5e72, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:36,135 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a29fa882a9e3f50c391e3aa08c5c5e72, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:55:36,135 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34681] ipc.CallRunner(144): callId: 54 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:40234 deadline: 1681527396132, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a29fa882a9e3f50c391e3aa08c5c5e72, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:36,135 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a29fa882a9e3f50c391e3aa08c5c5e72, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:55:36,135 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34681] ipc.CallRunner(144): callId: 55 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:40210 deadline: 1681527396133, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a29fa882a9e3f50c391e3aa08c5c5e72, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:36,140 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a29fa882a9e3f50c391e3aa08c5c5e72, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:55:36,141 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34681] ipc.CallRunner(144): callId: 57 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:40158 deadline: 1681527396137, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a29fa882a9e3f50c391e3aa08c5c5e72, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:36,141 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a29fa882a9e3f50c391e3aa08c5c5e72, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:55:36,141 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34681] ipc.CallRunner(144): callId: 58 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:40212 deadline: 1681527396137, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a29fa882a9e3f50c391e3aa08c5c5e72, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:36,233 INFO [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=106 (bloomFilter=true), to=hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/a29fa882a9e3f50c391e3aa08c5c5e72/.tmp/B/2df1e3eb809d43c0855e5c6a84429a07 2023-04-15 02:55:36,333 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a29fa882a9e3f50c391e3aa08c5c5e72, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:55:36,333 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34681] ipc.CallRunner(144): callId: 58 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:40192 deadline: 1681527396330, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a29fa882a9e3f50c391e3aa08c5c5e72, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:36,340 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a29fa882a9e3f50c391e3aa08c5c5e72, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:55:36,340 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34681] ipc.CallRunner(144): callId: 57 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:40210 deadline: 1681527396336, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a29fa882a9e3f50c391e3aa08c5c5e72, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:36,341 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a29fa882a9e3f50c391e3aa08c5c5e72, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:55:36,341 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34681] ipc.CallRunner(144): callId: 56 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:40234 deadline: 1681527396336, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a29fa882a9e3f50c391e3aa08c5c5e72, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:36,345 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a29fa882a9e3f50c391e3aa08c5c5e72, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:55:36,345 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34681] ipc.CallRunner(144): callId: 59 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:40158 deadline: 1681527396341, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a29fa882a9e3f50c391e3aa08c5c5e72, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:36,347 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a29fa882a9e3f50c391e3aa08c5c5e72, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:55:36,348 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34681] ipc.CallRunner(144): callId: 60 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:40212 deadline: 1681527396342, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a29fa882a9e3f50c391e3aa08c5c5e72, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:36,637 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a29fa882a9e3f50c391e3aa08c5c5e72, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:55:36,637 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34681] ipc.CallRunner(144): callId: 60 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:40192 deadline: 1681527396634, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a29fa882a9e3f50c391e3aa08c5c5e72, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:36,643 INFO [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=106 (bloomFilter=true), to=hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/a29fa882a9e3f50c391e3aa08c5c5e72/.tmp/C/10f17357ba8c428a8595c4e8b9454a50 2023-04-15 02:55:36,645 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a29fa882a9e3f50c391e3aa08c5c5e72, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:55:36,645 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34681] ipc.CallRunner(144): callId: 59 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:40210 deadline: 1681527396642, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a29fa882a9e3f50c391e3aa08c5c5e72, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:36,646 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/a29fa882a9e3f50c391e3aa08c5c5e72/.tmp/A/02ceb979dd0b4fde839b9ab0388624be as hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/a29fa882a9e3f50c391e3aa08c5c5e72/A/02ceb979dd0b4fde839b9ab0388624be 2023-04-15 02:55:36,647 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a29fa882a9e3f50c391e3aa08c5c5e72, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:55:36,647 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34681] ipc.CallRunner(144): callId: 58 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:40234 deadline: 1681527396644, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a29fa882a9e3f50c391e3aa08c5c5e72, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:36,649 INFO [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/a29fa882a9e3f50c391e3aa08c5c5e72/A/02ceb979dd0b4fde839b9ab0388624be, entries=150, sequenceid=106, filesize=11.6 K 2023-04-15 02:55:36,650 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/a29fa882a9e3f50c391e3aa08c5c5e72/.tmp/B/2df1e3eb809d43c0855e5c6a84429a07 as hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/a29fa882a9e3f50c391e3aa08c5c5e72/B/2df1e3eb809d43c0855e5c6a84429a07 2023-04-15 02:55:36,656 INFO [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/a29fa882a9e3f50c391e3aa08c5c5e72/B/2df1e3eb809d43c0855e5c6a84429a07, entries=150, sequenceid=106, filesize=11.6 K 2023-04-15 02:55:36,656 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a29fa882a9e3f50c391e3aa08c5c5e72, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:55:36,656 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34681] ipc.CallRunner(144): callId: 61 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:40158 deadline: 1681527396646, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a29fa882a9e3f50c391e3aa08c5c5e72, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:36,656 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/a29fa882a9e3f50c391e3aa08c5c5e72/.tmp/C/10f17357ba8c428a8595c4e8b9454a50 as hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/a29fa882a9e3f50c391e3aa08c5c5e72/C/10f17357ba8c428a8595c4e8b9454a50 2023-04-15 02:55:36,658 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a29fa882a9e3f50c391e3aa08c5c5e72, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:55:36,658 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34681] ipc.CallRunner(144): callId: 62 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:40212 deadline: 1681527396649, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a29fa882a9e3f50c391e3aa08c5c5e72, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:36,659 INFO [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/a29fa882a9e3f50c391e3aa08c5c5e72/C/10f17357ba8c428a8595c4e8b9454a50, entries=150, sequenceid=106, filesize=11.6 K 2023-04-15 02:55:36,660 INFO [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=127.47 KB/130530 for a29fa882a9e3f50c391e3aa08c5c5e72 in 846ms, sequenceid=106, compaction requested=true 2023-04-15 02:55:36,660 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for a29fa882a9e3f50c391e3aa08c5c5e72: 2023-04-15 02:55:36,660 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681527331158.a29fa882a9e3f50c391e3aa08c5c5e72. 2023-04-15 02:55:36,660 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-15 02:55:36,660 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-15 02:55:36,660 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:36,660 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-15 02:55:36,662 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34681,1681527209133' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-15 02:55:36,663 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:36,663 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:36,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:36,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:55:36,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:55:36,663 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:36,663 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-15 02:55:36,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:55:36,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:55:36,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:55:36,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:36,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:55:36,664 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34681,1681527209133' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-15 02:55:36,664 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-15 02:55:36,664 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@57c86543[Count = 0] remaining members to acquire global barrier 2023-04-15 02:55:36,664 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:36,665 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:36,665 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:36,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:36,665 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-15 02:55:36,665 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-15 02:55:36,665 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34681,1681527209133' in zk 2023-04-15 02:55:36,665 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:36,665 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-15 02:55:36,666 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-15 02:55:36,666 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:36,666 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-15 02:55:36,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:36,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:55:36,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:55:36,666 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-15 02:55:36,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:55:36,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:55:36,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:55:36,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:36,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:55:36,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:55:36,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:36,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34681,1681527209133': 2023-04-15 02:55:36,668 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34681,1681527209133' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-15 02:55:36,668 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-15 02:55:36,668 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-15 02:55:36,668 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-15 02:55:36,668 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-15 02:55:36,668 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-15 02:55:36,669 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:36,669 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:36,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:36,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:55:36,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:55:36,669 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-15 02:55:36,669 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:36,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:36,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:55:36,669 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:36,669 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-15 02:55:36,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-15 02:55:36,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:55:36,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:36,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:55:36,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:55:36,670 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:36,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:36,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:55:36,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:55:36,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:36,672 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:36,672 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:55:36,672 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:55:36,672 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:36,672 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-15 02:55:36,672 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-15 02:55:36,672 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-15 02:55:36,672 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-15 02:55:36,672 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-15 02:55:36,672 DEBUG [Thread-2739] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-15 02:55:36,672 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:36,672 DEBUG [Thread-2739] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-15 02:55:36,672 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-15 02:55:36,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-15 02:55:36,672 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-15 02:55:36,672 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:36,672 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:36,672 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-15 02:55:36,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-15 02:55:36,772 DEBUG [Thread-2739] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-15 02:55:36,773 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-15 02:55:36,774 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-15 02:55:36,775 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-15 02:55:36,775 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-15 02:55:36,775 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-15 02:55:36,775 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-15 02:55:36,775 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-15 02:55:36,775 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-15 02:55:36,775 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:36,775 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:55:36,776 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-15 02:55:36,776 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:36,776 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-15 02:55:36,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-15 02:55:36,776 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:36,776 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-15 02:55:36,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:55:36,776 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:36,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-15 02:55:36,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:55:36,777 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-15 02:55:36,777 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-15 02:55:36,777 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-15 02:55:36,777 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-15 02:55:36,777 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-15 02:55:36,777 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-15 02:55:36,777 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-15 02:55:36,777 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-15 02:55:36,777 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681527331158.a29fa882a9e3f50c391e3aa08c5c5e72. 2023-04-15 02:55:36,777 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681527331158.a29fa882a9e3f50c391e3aa08c5c5e72. started... 2023-04-15 02:55:36,777 INFO [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing a29fa882a9e3f50c391e3aa08c5c5e72 3/3 column families, dataSize=127.47 KB heapSize=365.06 KB 2023-04-15 02:55:36,778 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a29fa882a9e3f50c391e3aa08c5c5e72, store=A 2023-04-15 02:55:36,778 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-15 02:55:36,778 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a29fa882a9e3f50c391e3aa08c5c5e72, store=B 2023-04-15 02:55:36,778 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-15 02:55:36,778 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a29fa882a9e3f50c391e3aa08c5c5e72, store=C 2023-04-15 02:55:36,778 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-15 02:55:37,143 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34681] regionserver.HRegion(9158): Flush requested on a29fa882a9e3f50c391e3aa08c5c5e72 2023-04-15 02:55:37,143 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681527331158.a29fa882a9e3f50c391e3aa08c5c5e72. as already flushing 2023-04-15 02:55:37,162 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a29fa882a9e3f50c391e3aa08c5c5e72, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:55:37,162 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34681] ipc.CallRunner(144): callId: 63 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:40210 deadline: 1681527397157, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a29fa882a9e3f50c391e3aa08c5c5e72, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:37,163 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a29fa882a9e3f50c391e3aa08c5c5e72, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:55:37,163 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34681] ipc.CallRunner(144): callId: 62 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:40234 deadline: 1681527397160, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a29fa882a9e3f50c391e3aa08c5c5e72, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:37,165 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a29fa882a9e3f50c391e3aa08c5c5e72, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:55:37,165 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34681] ipc.CallRunner(144): callId: 63 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:40158 deadline: 1681527397161, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a29fa882a9e3f50c391e3aa08c5c5e72, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:37,166 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a29fa882a9e3f50c391e3aa08c5c5e72, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:55:37,166 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34681] ipc.CallRunner(144): callId: 64 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:40212 deadline: 1681527397161, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a29fa882a9e3f50c391e3aa08c5c5e72, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:37,168 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a29fa882a9e3f50c391e3aa08c5c5e72, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:55:37,169 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34681] ipc.CallRunner(144): callId: 66 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:40192 deadline: 1681527397162, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a29fa882a9e3f50c391e3aa08c5c5e72, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:37,186 INFO [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=142 (bloomFilter=true), to=hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/a29fa882a9e3f50c391e3aa08c5c5e72/.tmp/A/b802fc417bb640f5bd6179acfd948549 2023-04-15 02:55:37,199 INFO [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=142 (bloomFilter=true), to=hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/a29fa882a9e3f50c391e3aa08c5c5e72/.tmp/B/20e1b005b5354aee8174fe0b7cac8965 2023-04-15 02:55:37,206 INFO [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=142 (bloomFilter=true), to=hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/a29fa882a9e3f50c391e3aa08c5c5e72/.tmp/C/c6ed2ac20dd844d894c05929f3482006 2023-04-15 02:55:37,209 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/a29fa882a9e3f50c391e3aa08c5c5e72/.tmp/A/b802fc417bb640f5bd6179acfd948549 as hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/a29fa882a9e3f50c391e3aa08c5c5e72/A/b802fc417bb640f5bd6179acfd948549 2023-04-15 02:55:37,212 INFO [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/a29fa882a9e3f50c391e3aa08c5c5e72/A/b802fc417bb640f5bd6179acfd948549, entries=150, sequenceid=142, filesize=11.8 K 2023-04-15 02:55:37,213 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/a29fa882a9e3f50c391e3aa08c5c5e72/.tmp/B/20e1b005b5354aee8174fe0b7cac8965 as hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/a29fa882a9e3f50c391e3aa08c5c5e72/B/20e1b005b5354aee8174fe0b7cac8965 2023-04-15 02:55:37,216 INFO [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/a29fa882a9e3f50c391e3aa08c5c5e72/B/20e1b005b5354aee8174fe0b7cac8965, entries=150, sequenceid=142, filesize=11.8 K 2023-04-15 02:55:37,220 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/a29fa882a9e3f50c391e3aa08c5c5e72/.tmp/C/c6ed2ac20dd844d894c05929f3482006 as hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/a29fa882a9e3f50c391e3aa08c5c5e72/C/c6ed2ac20dd844d894c05929f3482006 2023-04-15 02:55:37,226 INFO [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/a29fa882a9e3f50c391e3aa08c5c5e72/C/c6ed2ac20dd844d894c05929f3482006, entries=150, sequenceid=142, filesize=11.8 K 2023-04-15 02:55:37,227 INFO [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~127.47 KB/130530, heapSize ~365.02 KB/373776, currentSize=53.67 KB/54960 for a29fa882a9e3f50c391e3aa08c5c5e72 in 450ms, sequenceid=142, compaction requested=true 2023-04-15 02:55:37,227 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for a29fa882a9e3f50c391e3aa08c5c5e72: 2023-04-15 02:55:37,227 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681527331158.a29fa882a9e3f50c391e3aa08c5c5e72. 2023-04-15 02:55:37,230 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-15 02:55:37,230 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-15 02:55:37,230 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:37,230 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-15 02:55:37,230 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34681,1681527209133' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-15 02:55:37,232 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:37,232 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:37,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:37,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:55:37,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:55:37,232 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:37,232 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-15 02:55:37,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:55:37,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:55:37,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:55:37,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:37,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:55:37,233 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34681,1681527209133' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-15 02:55:37,233 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-15 02:55:37,233 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6501bca6[Count = 0] remaining members to acquire global barrier 2023-04-15 02:55:37,233 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:37,234 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:37,234 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:37,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:37,234 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:37,234 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-15 02:55:37,234 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-15 02:55:37,234 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-15 02:55:37,234 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34681,1681527209133' in zk 2023-04-15 02:55:37,235 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:37,235 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-15 02:55:37,235 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-15 02:55:37,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:37,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:55:37,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:55:37,235 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-15 02:55:37,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:55:37,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:55:37,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:55:37,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:37,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:55:37,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:55:37,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:37,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34681,1681527209133': 2023-04-15 02:55:37,236 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34681,1681527209133' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-15 02:55:37,236 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-15 02:55:37,236 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-15 02:55:37,237 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-15 02:55:37,237 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-15 02:55:37,237 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-15 02:55:37,237 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:37,237 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-15 02:55:37,238 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:37,237 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:37,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:37,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:37,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:55:37,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:55:37,238 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-15 02:55:37,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-15 02:55:37,238 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:37,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:37,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:55:37,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:55:37,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:55:37,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:55:37,239 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:37,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:37,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:55:37,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:55:37,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:37,240 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-15 02:55:37,241 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-15 02:55:37,241 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-15 02:55:37,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-15 02:55:37,241 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-15 02:55:37,241 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-15 02:55:37,241 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-15 02:55:37,241 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-15 02:55:37,241 DEBUG [Thread-2739] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-15 02:55:37,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-15 02:55:37,241 DEBUG [Thread-2739] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-15 02:55:37,241 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:37,241 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:55:37,241 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:55:37,241 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:37,241 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:37,241 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:37,241 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:37,265 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34681] regionserver.HRegion(9158): Flush requested on a29fa882a9e3f50c391e3aa08c5c5e72 2023-04-15 02:55:37,265 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing a29fa882a9e3f50c391e3aa08c5c5e72 3/3 column families, dataSize=60.38 KB heapSize=180.05 KB 2023-04-15 02:55:37,265 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a29fa882a9e3f50c391e3aa08c5c5e72, store=A 2023-04-15 02:55:37,265 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-15 02:55:37,265 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a29fa882a9e3f50c391e3aa08c5c5e72, store=B 2023-04-15 02:55:37,265 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-15 02:55:37,265 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a29fa882a9e3f50c391e3aa08c5c5e72, store=C 2023-04-15 02:55:37,265 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-15 02:55:37,316 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a29fa882a9e3f50c391e3aa08c5c5e72, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:55:37,316 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34681] ipc.CallRunner(144): callId: 70 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:40234 deadline: 1681527397306, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a29fa882a9e3f50c391e3aa08c5c5e72, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:37,316 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a29fa882a9e3f50c391e3aa08c5c5e72, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:55:37,316 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34681] ipc.CallRunner(144): callId: 72 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:40210 deadline: 1681527397306, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a29fa882a9e3f50c391e3aa08c5c5e72, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:37,316 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a29fa882a9e3f50c391e3aa08c5c5e72, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:55:37,317 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34681] ipc.CallRunner(144): callId: 73 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:40192 deadline: 1681527397312, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a29fa882a9e3f50c391e3aa08c5c5e72, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:37,341 DEBUG [Thread-2739] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-15 02:55:37,342 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-15 02:55:37,343 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-15 02:55:37,344 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-15 02:55:37,344 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-15 02:55:37,344 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-15 02:55:37,344 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-15 02:55:37,344 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-15 02:55:37,344 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-15 02:55:37,344 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:37,344 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:55:37,345 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-15 02:55:37,345 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:37,345 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-15 02:55:37,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-15 02:55:37,345 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:37,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:55:37,345 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-15 02:55:37,346 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:37,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-15 02:55:37,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:55:37,346 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-15 02:55:37,348 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-15 02:55:37,348 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-15 02:55:37,348 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-15 02:55:37,348 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-15 02:55:37,348 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-15 02:55:37,348 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-15 02:55:37,348 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-15 02:55:37,348 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681527331158.a29fa882a9e3f50c391e3aa08c5c5e72. 2023-04-15 02:55:37,348 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681527331158.a29fa882a9e3f50c391e3aa08c5c5e72. started... 2023-04-15 02:55:37,348 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681527331158.a29fa882a9e3f50c391e3aa08c5c5e72. as already flushing 2023-04-15 02:55:37,348 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681527331158.a29fa882a9e3f50c391e3aa08c5c5e72. 2023-04-15 02:55:37,348 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-15 02:55:37,348 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-15 02:55:37,348 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:37,348 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-15 02:55:37,348 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34681,1681527209133' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-15 02:55:37,349 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:37,349 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:37,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:37,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:55:37,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:55:37,349 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:37,349 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-15 02:55:37,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:55:37,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:55:37,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:55:37,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:37,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:55:37,350 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34681,1681527209133' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-15 02:55:37,351 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-15 02:55:37,351 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@41563ed9[Count = 0] remaining members to acquire global barrier 2023-04-15 02:55:37,351 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:37,351 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:37,351 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-15 02:55:37,352 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:37,352 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:37,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:37,352 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-15 02:55:37,352 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-15 02:55:37,352 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34681,1681527209133' in zk 2023-04-15 02:55:37,353 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-15 02:55:37,353 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:37,353 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-15 02:55:37,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:37,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:55:37,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:55:37,353 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-15 02:55:37,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:55:37,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:55:37,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:55:37,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:37,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:55:37,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:55:37,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:37,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34681,1681527209133': 2023-04-15 02:55:37,355 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34681,1681527209133' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-15 02:55:37,355 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-15 02:55:37,355 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-15 02:55:37,355 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-15 02:55:37,355 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-15 02:55:37,355 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-15 02:55:37,355 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:37,356 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:37,356 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:37,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:37,356 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-15 02:55:37,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:37,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:55:37,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:55:37,356 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-15 02:55:37,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-15 02:55:37,356 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:37,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:55:37,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:37,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:55:37,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:55:37,357 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:37,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:55:37,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:37,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:55:37,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:55:37,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:37,358 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-15 02:55:37,358 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-15 02:55:37,358 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-15 02:55:37,358 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-15 02:55:37,358 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-15 02:55:37,358 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-15 02:55:37,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-15 02:55:37,359 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:37,359 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:55:37,359 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:55:37,359 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:37,359 DEBUG [Thread-2739] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-15 02:55:37,359 DEBUG [Thread-2739] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-15 02:55:37,359 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:37,359 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:37,359 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:37,359 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-15 02:55:37,359 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-15 02:55:37,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-15 02:55:37,421 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a29fa882a9e3f50c391e3aa08c5c5e72, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:55:37,421 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34681] ipc.CallRunner(144): callId: 72 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:40234 deadline: 1681527397417, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a29fa882a9e3f50c391e3aa08c5c5e72, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:37,421 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a29fa882a9e3f50c391e3aa08c5c5e72, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:55:37,421 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34681] ipc.CallRunner(144): callId: 74 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:40210 deadline: 1681527397417, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a29fa882a9e3f50c391e3aa08c5c5e72, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:37,422 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a29fa882a9e3f50c391e3aa08c5c5e72, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:55:37,422 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34681] ipc.CallRunner(144): callId: 75 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:40192 deadline: 1681527397417, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a29fa882a9e3f50c391e3aa08c5c5e72, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:37,459 DEBUG [Thread-2739] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-15 02:55:37,460 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-15 02:55:37,461 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-15 02:55:37,461 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-15 02:55:37,461 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-15 02:55:37,461 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-15 02:55:37,462 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-15 02:55:37,462 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-15 02:55:37,462 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-15 02:55:37,462 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:37,462 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:55:37,462 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:37,462 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-15 02:55:37,463 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-15 02:55:37,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-15 02:55:37,463 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:37,463 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-15 02:55:37,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:55:37,463 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:37,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-15 02:55:37,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:55:37,463 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-15 02:55:37,464 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-15 02:55:37,464 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-15 02:55:37,464 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-15 02:55:37,464 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-15 02:55:37,464 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-15 02:55:37,464 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-15 02:55:37,464 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-15 02:55:37,464 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681527331158.a29fa882a9e3f50c391e3aa08c5c5e72. 2023-04-15 02:55:37,465 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681527331158.a29fa882a9e3f50c391e3aa08c5c5e72. started... 2023-04-15 02:55:37,465 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681527331158.a29fa882a9e3f50c391e3aa08c5c5e72. as already flushing 2023-04-15 02:55:37,465 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681527331158.a29fa882a9e3f50c391e3aa08c5c5e72. 2023-04-15 02:55:37,465 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-15 02:55:37,465 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-15 02:55:37,465 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:37,465 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-15 02:55:37,465 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34681,1681527209133' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-15 02:55:37,466 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:37,466 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:37,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:37,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:55:37,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:55:37,466 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:37,466 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-15 02:55:37,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:55:37,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:55:37,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:55:37,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:37,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:55:37,470 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34681,1681527209133' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-15 02:55:37,470 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@18e5e6e1[Count = 0] remaining members to acquire global barrier 2023-04-15 02:55:37,470 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-15 02:55:37,470 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:37,479 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:37,479 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:37,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:37,479 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-15 02:55:37,479 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-15 02:55:37,479 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:37,479 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-15 02:55:37,479 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34681,1681527209133' in zk 2023-04-15 02:55:37,480 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:37,480 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-15 02:55:37,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:37,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:55:37,480 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-15 02:55:37,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:55:37,480 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-15 02:55:37,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:55:37,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:55:37,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:55:37,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:37,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:55:37,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:55:37,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:37,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34681,1681527209133': 2023-04-15 02:55:37,482 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-15 02:55:37,482 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-15 02:55:37,482 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34681,1681527209133' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-15 02:55:37,482 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-15 02:55:37,482 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-15 02:55:37,482 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-15 02:55:37,483 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:37,483 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:37,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:37,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:55:37,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:55:37,483 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:37,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:37,483 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-15 02:55:37,483 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:37,483 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-15 02:55:37,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-15 02:55:37,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:55:37,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:37,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:55:37,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:55:37,484 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:37,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:55:37,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:37,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:55:37,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:55:37,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:37,487 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-15 02:55:37,487 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:37,487 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-15 02:55:37,487 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-15 02:55:37,487 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-15 02:55:37,487 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-15 02:55:37,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-15 02:55:37,487 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-15 02:55:37,487 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-15 02:55:37,487 DEBUG [Thread-2739] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-15 02:55:37,487 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:55:37,488 DEBUG [Thread-2739] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-15 02:55:37,488 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:55:37,488 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-15 02:55:37,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-15 02:55:37,488 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:37,488 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:37,488 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:37,488 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:37,588 DEBUG [Thread-2739] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-15 02:55:37,588 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-15 02:55:37,589 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-15 02:55:37,590 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-15 02:55:37,590 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-15 02:55:37,590 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-15 02:55:37,590 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-15 02:55:37,590 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-15 02:55:37,590 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-15 02:55:37,591 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:37,591 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:55:37,591 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:37,591 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-15 02:55:37,591 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-15 02:55:37,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-15 02:55:37,591 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:37,592 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-15 02:55:37,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:55:37,592 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:37,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-15 02:55:37,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:55:37,592 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-15 02:55:37,593 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-15 02:55:37,593 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-15 02:55:37,593 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-15 02:55:37,593 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-15 02:55:37,593 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-15 02:55:37,593 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-15 02:55:37,593 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-15 02:55:37,593 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681527331158.a29fa882a9e3f50c391e3aa08c5c5e72. 2023-04-15 02:55:37,593 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681527331158.a29fa882a9e3f50c391e3aa08c5c5e72. started... 2023-04-15 02:55:37,593 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681527331158.a29fa882a9e3f50c391e3aa08c5c5e72. as already flushing 2023-04-15 02:55:37,593 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681527331158.a29fa882a9e3f50c391e3aa08c5c5e72. 2023-04-15 02:55:37,593 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-15 02:55:37,593 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-15 02:55:37,593 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:37,593 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-15 02:55:37,593 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34681,1681527209133' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-15 02:55:37,594 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:37,594 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:37,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:37,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:55:37,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:55:37,594 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:37,594 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-15 02:55:37,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:55:37,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:55:37,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:55:37,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:37,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:55:37,595 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34681,1681527209133' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-15 02:55:37,595 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-15 02:55:37,595 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@c937ac3[Count = 0] remaining members to acquire global barrier 2023-04-15 02:55:37,595 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:37,596 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:37,596 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:37,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:37,596 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-15 02:55:37,596 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-15 02:55:37,596 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34681,1681527209133' in zk 2023-04-15 02:55:37,596 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:37,596 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-15 02:55:37,597 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-15 02:55:37,597 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-15 02:55:37,597 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-15 02:55:37,597 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:37,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:37,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:55:37,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:55:37,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:55:37,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:55:37,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:55:37,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:37,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:55:37,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:55:37,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:37,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34681,1681527209133': 2023-04-15 02:55:37,599 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34681,1681527209133' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-15 02:55:37,599 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-15 02:55:37,599 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-15 02:55:37,599 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-15 02:55:37,599 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-15 02:55:37,599 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-15 02:55:37,601 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:37,601 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:37,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:37,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:55:37,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:55:37,601 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:37,601 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-15 02:55:37,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:37,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:55:37,602 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-15 02:55:37,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-15 02:55:37,602 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:37,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:55:37,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:37,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:55:37,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:55:37,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:37,603 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:37,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:55:37,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:55:37,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:37,604 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-15 02:55:37,604 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-15 02:55:37,604 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-15 02:55:37,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-15 02:55:37,604 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-15 02:55:37,604 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:37,604 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-15 02:55:37,604 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-15 02:55:37,604 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-15 02:55:37,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-15 02:55:37,604 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:55:37,604 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:55:37,605 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:37,605 DEBUG [Thread-2739] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-15 02:55:37,605 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:37,605 DEBUG [Thread-2739] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-15 02:55:37,605 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:37,605 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:37,630 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a29fa882a9e3f50c391e3aa08c5c5e72, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:55:37,631 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34681] ipc.CallRunner(144): callId: 76 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:40210 deadline: 1681527397622, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a29fa882a9e3f50c391e3aa08c5c5e72, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:37,631 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a29fa882a9e3f50c391e3aa08c5c5e72, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:55:37,631 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34681] ipc.CallRunner(144): callId: 74 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:40234 deadline: 1681527397622, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a29fa882a9e3f50c391e3aa08c5c5e72, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:37,631 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a29fa882a9e3f50c391e3aa08c5c5e72, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:55:37,631 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34681] ipc.CallRunner(144): callId: 77 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:40192 deadline: 1681527397622, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a29fa882a9e3f50c391e3aa08c5c5e72, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:37,675 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=154 (bloomFilter=true), to=hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/a29fa882a9e3f50c391e3aa08c5c5e72/.tmp/A/b0e063b9c503473580ad242c3fc542ea 2023-04-15 02:55:37,694 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=154 (bloomFilter=true), to=hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/a29fa882a9e3f50c391e3aa08c5c5e72/.tmp/B/6af0f1b08cef4b61889eb01e2a3de70b 2023-04-15 02:55:37,705 DEBUG [Thread-2739] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-15 02:55:37,705 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-15 02:55:37,707 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-15 02:55:37,708 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-15 02:55:37,708 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=154 (bloomFilter=true), to=hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/a29fa882a9e3f50c391e3aa08c5c5e72/.tmp/C/b871420ae8d8488395306a0bb12057fb 2023-04-15 02:55:37,708 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-15 02:55:37,708 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-15 02:55:37,708 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-15 02:55:37,709 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-15 02:55:37,709 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-15 02:55:37,709 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:37,709 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:55:37,710 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-15 02:55:37,710 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-15 02:55:37,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-15 02:55:37,710 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:37,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:55:37,710 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:37,710 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-15 02:55:37,711 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:37,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-15 02:55:37,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:55:37,711 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-15 02:55:37,711 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-15 02:55:37,711 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-15 02:55:37,711 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-15 02:55:37,711 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-15 02:55:37,711 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-15 02:55:37,711 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-15 02:55:37,712 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-15 02:55:37,712 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681527331158.a29fa882a9e3f50c391e3aa08c5c5e72. 2023-04-15 02:55:37,712 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681527331158.a29fa882a9e3f50c391e3aa08c5c5e72. started... 2023-04-15 02:55:37,712 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681527331158.a29fa882a9e3f50c391e3aa08c5c5e72. as already flushing 2023-04-15 02:55:37,712 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681527331158.a29fa882a9e3f50c391e3aa08c5c5e72. 2023-04-15 02:55:37,712 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/a29fa882a9e3f50c391e3aa08c5c5e72/.tmp/A/b0e063b9c503473580ad242c3fc542ea as hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/a29fa882a9e3f50c391e3aa08c5c5e72/A/b0e063b9c503473580ad242c3fc542ea 2023-04-15 02:55:37,712 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-15 02:55:37,712 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-15 02:55:37,712 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:37,712 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-15 02:55:37,712 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34681,1681527209133' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-15 02:55:37,713 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:37,713 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:37,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:37,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:55:37,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:55:37,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:55:37,713 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:37,713 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-15 02:55:37,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:55:37,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:55:37,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:37,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:55:37,715 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/a29fa882a9e3f50c391e3aa08c5c5e72/A/b0e063b9c503473580ad242c3fc542ea, entries=200, sequenceid=154, filesize=14.1 K 2023-04-15 02:55:37,716 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/a29fa882a9e3f50c391e3aa08c5c5e72/.tmp/B/6af0f1b08cef4b61889eb01e2a3de70b as hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/a29fa882a9e3f50c391e3aa08c5c5e72/B/6af0f1b08cef4b61889eb01e2a3de70b 2023-04-15 02:55:37,716 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34681,1681527209133' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-15 02:55:37,716 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@649f037b[Count = 0] remaining members to acquire global barrier 2023-04-15 02:55:37,716 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-15 02:55:37,716 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:37,717 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:37,717 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:37,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:37,717 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-15 02:55:37,717 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-15 02:55:37,717 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34681,1681527209133' in zk 2023-04-15 02:55:37,717 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:37,717 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-15 02:55:37,718 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:37,718 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-15 02:55:37,718 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-15 02:55:37,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:37,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:55:37,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:55:37,718 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-15 02:55:37,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:55:37,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:55:37,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:55:37,719 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/a29fa882a9e3f50c391e3aa08c5c5e72/B/6af0f1b08cef4b61889eb01e2a3de70b, entries=150, sequenceid=154, filesize=11.8 K 2023-04-15 02:55:37,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:37,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:55:37,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:55:37,719 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/a29fa882a9e3f50c391e3aa08c5c5e72/.tmp/C/b871420ae8d8488395306a0bb12057fb as hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/a29fa882a9e3f50c391e3aa08c5c5e72/C/b871420ae8d8488395306a0bb12057fb 2023-04-15 02:55:37,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:37,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34681,1681527209133': 2023-04-15 02:55:37,720 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34681,1681527209133' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-15 02:55:37,720 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-15 02:55:37,720 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-15 02:55:37,720 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-15 02:55:37,720 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-15 02:55:37,720 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-15 02:55:37,721 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:37,721 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:37,721 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:37,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:37,721 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-15 02:55:37,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:37,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:55:37,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:55:37,721 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-15 02:55:37,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-15 02:55:37,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:37,722 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:37,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:55:37,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:55:37,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:55:37,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:55:37,723 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:37,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:37,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:55:37,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:55:37,724 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/a29fa882a9e3f50c391e3aa08c5c5e72/C/b871420ae8d8488395306a0bb12057fb, entries=150, sequenceid=154, filesize=11.8 K 2023-04-15 02:55:37,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:37,724 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~60.38 KB/61830, heapSize ~180 KB/184320, currentSize=114.05 KB/116790 for a29fa882a9e3f50c391e3aa08c5c5e72 in 459ms, sequenceid=154, compaction requested=true 2023-04-15 02:55:37,724 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for a29fa882a9e3f50c391e3aa08c5c5e72: 2023-04-15 02:55:37,724 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:37,724 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:55:37,725 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:55:37,725 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-15 02:55:37,725 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-15 02:55:37,725 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-15 02:55:37,725 DEBUG [RS:0;jenkins-hbase20:34681-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-04-15 02:55:37,725 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-15 02:55:37,725 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-15 02:55:37,725 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:37,725 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-15 02:55:37,725 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:37,725 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-15 02:55:37,725 DEBUG [Thread-2739] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-15 02:55:37,725 DEBUG [Thread-2739] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-15 02:55:37,725 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-15 02:55:37,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-15 02:55:37,725 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-15 02:55:37,725 DEBUG [RS:0;jenkins-hbase20:34681-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-04-15 02:55:37,725 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:37,725 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-15 02:55:37,725 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:37,725 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-15 02:55:37,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-15 02:55:37,727 DEBUG [RS:0;jenkins-hbase20:34681-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 62223 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-04-15 02:55:37,727 DEBUG [RS:0;jenkins-hbase20:34681-shortCompactions-0] regionserver.HStore(1912): a29fa882a9e3f50c391e3aa08c5c5e72/A is initiating minor compaction (all files) 2023-04-15 02:55:37,727 INFO [RS:0;jenkins-hbase20:34681-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of a29fa882a9e3f50c391e3aa08c5c5e72/A in TestAcidGuarantees,,1681527331158.a29fa882a9e3f50c391e3aa08c5c5e72. 2023-04-15 02:55:37,727 INFO [RS:0;jenkins-hbase20:34681-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/a29fa882a9e3f50c391e3aa08c5c5e72/A/ade7f87389884e4ea915a78f74f55f27, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/a29fa882a9e3f50c391e3aa08c5c5e72/A/bee8d060091749a4af47caf99098c5bb, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/a29fa882a9e3f50c391e3aa08c5c5e72/A/02ceb979dd0b4fde839b9ab0388624be, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/a29fa882a9e3f50c391e3aa08c5c5e72/A/b802fc417bb640f5bd6179acfd948549, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/a29fa882a9e3f50c391e3aa08c5c5e72/A/b0e063b9c503473580ad242c3fc542ea] into tmpdir=hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/a29fa882a9e3f50c391e3aa08c5c5e72/.tmp, totalSize=60.8 K 2023-04-15 02:55:37,728 DEBUG [RS:0;jenkins-hbase20:34681-shortCompactions-0] compactions.Compactor(207): Compacting ade7f87389884e4ea915a78f74f55f27, keycount=150, bloomtype=ROW, size=11.7 K, encoding=NONE, compression=NONE, seqNum=57, earliestPutTs=1681527334679 2023-04-15 02:55:37,728 DEBUG [RS:0;jenkins-hbase20:34681-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 59833 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-04-15 02:55:37,728 DEBUG [RS:0;jenkins-hbase20:34681-shortCompactions-0] compactions.Compactor(207): Compacting bee8d060091749a4af47caf99098c5bb, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=93, earliestPutTs=1681527334726 2023-04-15 02:55:37,728 DEBUG [RS:0;jenkins-hbase20:34681-longCompactions-0] regionserver.HStore(1912): a29fa882a9e3f50c391e3aa08c5c5e72/B is initiating minor compaction (all files) 2023-04-15 02:55:37,728 INFO [RS:0;jenkins-hbase20:34681-longCompactions-0] regionserver.HRegion(2259): Starting compaction of a29fa882a9e3f50c391e3aa08c5c5e72/B in TestAcidGuarantees,,1681527331158.a29fa882a9e3f50c391e3aa08c5c5e72. 2023-04-15 02:55:37,728 INFO [RS:0;jenkins-hbase20:34681-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/a29fa882a9e3f50c391e3aa08c5c5e72/B/e3ff06b400854006852039779a95aade, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/a29fa882a9e3f50c391e3aa08c5c5e72/B/c05c1b5850724aa8b501d9b7acd62a0c, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/a29fa882a9e3f50c391e3aa08c5c5e72/B/2df1e3eb809d43c0855e5c6a84429a07, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/a29fa882a9e3f50c391e3aa08c5c5e72/B/20e1b005b5354aee8174fe0b7cac8965, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/a29fa882a9e3f50c391e3aa08c5c5e72/B/6af0f1b08cef4b61889eb01e2a3de70b] into tmpdir=hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/a29fa882a9e3f50c391e3aa08c5c5e72/.tmp, totalSize=58.4 K 2023-04-15 02:55:37,728 DEBUG [RS:0;jenkins-hbase20:34681-shortCompactions-0] compactions.Compactor(207): Compacting 02ceb979dd0b4fde839b9ab0388624be, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=106, earliestPutTs=1681527335361 2023-04-15 02:55:37,729 DEBUG [RS:0;jenkins-hbase20:34681-longCompactions-0] compactions.Compactor(207): Compacting e3ff06b400854006852039779a95aade, keycount=150, bloomtype=ROW, size=11.7 K, encoding=NONE, compression=NONE, seqNum=57, earliestPutTs=1681527334679 2023-04-15 02:55:37,729 DEBUG [RS:0;jenkins-hbase20:34681-shortCompactions-0] compactions.Compactor(207): Compacting b802fc417bb640f5bd6179acfd948549, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=142, earliestPutTs=1681527336015 2023-04-15 02:55:37,729 DEBUG [RS:0;jenkins-hbase20:34681-longCompactions-0] compactions.Compactor(207): Compacting c05c1b5850724aa8b501d9b7acd62a0c, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=93, earliestPutTs=1681527334726 2023-04-15 02:55:37,729 DEBUG [RS:0;jenkins-hbase20:34681-shortCompactions-0] compactions.Compactor(207): Compacting b0e063b9c503473580ad242c3fc542ea, keycount=200, bloomtype=ROW, size=14.1 K, encoding=NONE, compression=NONE, seqNum=154, earliestPutTs=1681527337148 2023-04-15 02:55:37,730 DEBUG [RS:0;jenkins-hbase20:34681-longCompactions-0] compactions.Compactor(207): Compacting 2df1e3eb809d43c0855e5c6a84429a07, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=106, earliestPutTs=1681527335361 2023-04-15 02:55:37,730 DEBUG [RS:0;jenkins-hbase20:34681-longCompactions-0] compactions.Compactor(207): Compacting 20e1b005b5354aee8174fe0b7cac8965, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=142, earliestPutTs=1681527336015 2023-04-15 02:55:37,731 DEBUG [RS:0;jenkins-hbase20:34681-longCompactions-0] compactions.Compactor(207): Compacting 6af0f1b08cef4b61889eb01e2a3de70b, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=154, earliestPutTs=1681527337148 2023-04-15 02:55:37,750 INFO [RS:0;jenkins-hbase20:34681-longCompactions-0] throttle.PressureAwareThroughputController(145): a29fa882a9e3f50c391e3aa08c5c5e72#B#compaction#368 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-15 02:55:37,755 INFO [RS:0;jenkins-hbase20:34681-shortCompactions-0] throttle.PressureAwareThroughputController(145): a29fa882a9e3f50c391e3aa08c5c5e72#A#compaction#369 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-15 02:55:37,825 DEBUG [Thread-2739] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-15 02:55:37,826 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-15 02:55:37,827 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-15 02:55:37,828 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-15 02:55:37,828 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-15 02:55:37,828 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-15 02:55:37,828 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-15 02:55:37,828 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-15 02:55:37,828 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-15 02:55:37,828 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:37,828 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:55:37,829 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:37,829 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-15 02:55:37,829 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-15 02:55:37,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-15 02:55:37,829 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:37,829 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-15 02:55:37,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:55:37,829 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:37,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-15 02:55:37,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:55:37,830 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-15 02:55:37,830 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-15 02:55:37,830 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-15 02:55:37,830 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-15 02:55:37,830 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-15 02:55:37,830 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-15 02:55:37,830 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-15 02:55:37,830 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-15 02:55:37,830 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681527331158.a29fa882a9e3f50c391e3aa08c5c5e72. 2023-04-15 02:55:37,830 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681527331158.a29fa882a9e3f50c391e3aa08c5c5e72. started... 2023-04-15 02:55:37,830 INFO [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing a29fa882a9e3f50c391e3aa08c5c5e72 3/3 column families, dataSize=114.05 KB heapSize=339.42 KB 2023-04-15 02:55:37,830 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a29fa882a9e3f50c391e3aa08c5c5e72, store=A 2023-04-15 02:55:37,831 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-15 02:55:37,831 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a29fa882a9e3f50c391e3aa08c5c5e72, store=B 2023-04-15 02:55:37,831 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-15 02:55:37,831 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a29fa882a9e3f50c391e3aa08c5c5e72, store=C 2023-04-15 02:55:37,831 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-15 02:55:37,936 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34681] regionserver.HRegion(9158): Flush requested on a29fa882a9e3f50c391e3aa08c5c5e72 2023-04-15 02:55:37,936 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681527331158.a29fa882a9e3f50c391e3aa08c5c5e72. as already flushing 2023-04-15 02:55:37,975 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a29fa882a9e3f50c391e3aa08c5c5e72, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:55:37,975 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34681] ipc.CallRunner(144): callId: 81 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:40210 deadline: 1681527397969, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a29fa882a9e3f50c391e3aa08c5c5e72, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:37,978 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a29fa882a9e3f50c391e3aa08c5c5e72, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:55:37,978 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34681] ipc.CallRunner(144): callId: 79 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:40234 deadline: 1681527397970, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a29fa882a9e3f50c391e3aa08c5c5e72, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:37,978 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a29fa882a9e3f50c391e3aa08c5c5e72, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:55:37,978 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34681] ipc.CallRunner(144): callId: 82 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:40192 deadline: 1681527397971, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a29fa882a9e3f50c391e3aa08c5c5e72, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:38,078 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a29fa882a9e3f50c391e3aa08c5c5e72, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:55:38,078 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34681] ipc.CallRunner(144): callId: 83 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:40210 deadline: 1681527398076, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a29fa882a9e3f50c391e3aa08c5c5e72, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:38,082 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a29fa882a9e3f50c391e3aa08c5c5e72, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:55:38,082 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34681] ipc.CallRunner(144): callId: 81 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:40234 deadline: 1681527398078, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a29fa882a9e3f50c391e3aa08c5c5e72, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:38,085 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a29fa882a9e3f50c391e3aa08c5c5e72, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:55:38,086 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34681] ipc.CallRunner(144): callId: 84 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:40192 deadline: 1681527398079, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a29fa882a9e3f50c391e3aa08c5c5e72, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:38,168 DEBUG [RS:0;jenkins-hbase20:34681-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/a29fa882a9e3f50c391e3aa08c5c5e72/.tmp/B/8669bced50064421bc35c2bbf8ba2f27 as hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/a29fa882a9e3f50c391e3aa08c5c5e72/B/8669bced50064421bc35c2bbf8ba2f27 2023-04-15 02:55:38,169 DEBUG [RS:0;jenkins-hbase20:34681-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/a29fa882a9e3f50c391e3aa08c5c5e72/.tmp/A/566f729b22db4659ab9e6ff55d4bfc41 as hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/a29fa882a9e3f50c391e3aa08c5c5e72/A/566f729b22db4659ab9e6ff55d4bfc41 2023-04-15 02:55:38,171 INFO [RS:0;jenkins-hbase20:34681-longCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in a29fa882a9e3f50c391e3aa08c5c5e72/B of a29fa882a9e3f50c391e3aa08c5c5e72 into 8669bced50064421bc35c2bbf8ba2f27(size=12.0 K), total size for store is 12.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-15 02:55:38,172 DEBUG [RS:0;jenkins-hbase20:34681-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for a29fa882a9e3f50c391e3aa08c5c5e72: 2023-04-15 02:55:38,172 INFO [RS:0;jenkins-hbase20:34681-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681527331158.a29fa882a9e3f50c391e3aa08c5c5e72., storeName=a29fa882a9e3f50c391e3aa08c5c5e72/B, priority=11, startTime=1681527337725; duration=0sec 2023-04-15 02:55:38,172 DEBUG [RS:0;jenkins-hbase20:34681-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-15 02:55:38,172 DEBUG [RS:0;jenkins-hbase20:34681-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-04-15 02:55:38,176 DEBUG [RS:0;jenkins-hbase20:34681-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 59833 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-04-15 02:55:38,176 DEBUG [RS:0;jenkins-hbase20:34681-longCompactions-0] regionserver.HStore(1912): a29fa882a9e3f50c391e3aa08c5c5e72/C is initiating minor compaction (all files) 2023-04-15 02:55:38,176 INFO [RS:0;jenkins-hbase20:34681-longCompactions-0] regionserver.HRegion(2259): Starting compaction of a29fa882a9e3f50c391e3aa08c5c5e72/C in TestAcidGuarantees,,1681527331158.a29fa882a9e3f50c391e3aa08c5c5e72. 2023-04-15 02:55:38,176 INFO [RS:0;jenkins-hbase20:34681-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/a29fa882a9e3f50c391e3aa08c5c5e72/C/451fc6fe095f4e2f8ac779de5c121c27, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/a29fa882a9e3f50c391e3aa08c5c5e72/C/a55c129a820d464a8dd3cf851be769e5, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/a29fa882a9e3f50c391e3aa08c5c5e72/C/10f17357ba8c428a8595c4e8b9454a50, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/a29fa882a9e3f50c391e3aa08c5c5e72/C/c6ed2ac20dd844d894c05929f3482006, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/a29fa882a9e3f50c391e3aa08c5c5e72/C/b871420ae8d8488395306a0bb12057fb] into tmpdir=hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/a29fa882a9e3f50c391e3aa08c5c5e72/.tmp, totalSize=58.4 K 2023-04-15 02:55:38,179 DEBUG [RS:0;jenkins-hbase20:34681-longCompactions-0] compactions.Compactor(207): Compacting 451fc6fe095f4e2f8ac779de5c121c27, keycount=150, bloomtype=ROW, size=11.7 K, encoding=NONE, compression=NONE, seqNum=57, earliestPutTs=1681527334679 2023-04-15 02:55:38,179 DEBUG [RS:0;jenkins-hbase20:34681-longCompactions-0] compactions.Compactor(207): Compacting a55c129a820d464a8dd3cf851be769e5, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=93, earliestPutTs=1681527334726 2023-04-15 02:55:38,179 DEBUG [RS:0;jenkins-hbase20:34681-longCompactions-0] compactions.Compactor(207): Compacting 10f17357ba8c428a8595c4e8b9454a50, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=106, earliestPutTs=1681527335361 2023-04-15 02:55:38,180 INFO [RS:0;jenkins-hbase20:34681-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in a29fa882a9e3f50c391e3aa08c5c5e72/A of a29fa882a9e3f50c391e3aa08c5c5e72 into 566f729b22db4659ab9e6ff55d4bfc41(size=12.0 K), total size for store is 12.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-15 02:55:38,180 DEBUG [RS:0;jenkins-hbase20:34681-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for a29fa882a9e3f50c391e3aa08c5c5e72: 2023-04-15 02:55:38,180 INFO [RS:0;jenkins-hbase20:34681-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681527331158.a29fa882a9e3f50c391e3aa08c5c5e72., storeName=a29fa882a9e3f50c391e3aa08c5c5e72/A, priority=11, startTime=1681527337724; duration=0sec 2023-04-15 02:55:38,180 DEBUG [RS:0;jenkins-hbase20:34681-longCompactions-0] compactions.Compactor(207): Compacting c6ed2ac20dd844d894c05929f3482006, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=142, earliestPutTs=1681527336015 2023-04-15 02:55:38,180 DEBUG [RS:0;jenkins-hbase20:34681-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-15 02:55:38,180 DEBUG [RS:0;jenkins-hbase20:34681-longCompactions-0] compactions.Compactor(207): Compacting b871420ae8d8488395306a0bb12057fb, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=154, earliestPutTs=1681527337148 2023-04-15 02:55:38,182 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a29fa882a9e3f50c391e3aa08c5c5e72, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:55:38,182 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34681] ipc.CallRunner(144): callId: 65 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:40158 deadline: 1681527398167, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a29fa882a9e3f50c391e3aa08c5c5e72, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:38,182 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a29fa882a9e3f50c391e3aa08c5c5e72, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:55:38,182 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34681] ipc.CallRunner(144): callId: 66 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:40212 deadline: 1681527398168, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a29fa882a9e3f50c391e3aa08c5c5e72, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:38,185 INFO [RS:0;jenkins-hbase20:34681-longCompactions-0] throttle.PressureAwareThroughputController(145): a29fa882a9e3f50c391e3aa08c5c5e72#C#compaction#371 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-15 02:55:38,238 INFO [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=174 (bloomFilter=true), to=hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/a29fa882a9e3f50c391e3aa08c5c5e72/.tmp/A/e968895af35f4ae7a46c8b0e22f49d25 2023-04-15 02:55:38,295 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a29fa882a9e3f50c391e3aa08c5c5e72, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:55:38,295 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34681] ipc.CallRunner(144): callId: 86 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:40192 deadline: 1681527398289, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a29fa882a9e3f50c391e3aa08c5c5e72, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:38,295 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a29fa882a9e3f50c391e3aa08c5c5e72, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:55:38,296 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34681] ipc.CallRunner(144): callId: 85 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:40210 deadline: 1681527398289, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a29fa882a9e3f50c391e3aa08c5c5e72, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:38,296 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a29fa882a9e3f50c391e3aa08c5c5e72, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:55:38,296 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34681] ipc.CallRunner(144): callId: 83 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:40234 deadline: 1681527398289, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a29fa882a9e3f50c391e3aa08c5c5e72, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:38,594 DEBUG [RS:0;jenkins-hbase20:34681-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/a29fa882a9e3f50c391e3aa08c5c5e72/.tmp/C/7fdf48a747e543e1bc9b832166cde47a as hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/a29fa882a9e3f50c391e3aa08c5c5e72/C/7fdf48a747e543e1bc9b832166cde47a 2023-04-15 02:55:38,597 INFO [RS:0;jenkins-hbase20:34681-longCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in a29fa882a9e3f50c391e3aa08c5c5e72/C of a29fa882a9e3f50c391e3aa08c5c5e72 into 7fdf48a747e543e1bc9b832166cde47a(size=12.0 K), total size for store is 12.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-15 02:55:38,597 DEBUG [RS:0;jenkins-hbase20:34681-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for a29fa882a9e3f50c391e3aa08c5c5e72: 2023-04-15 02:55:38,597 INFO [RS:0;jenkins-hbase20:34681-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681527331158.a29fa882a9e3f50c391e3aa08c5c5e72., storeName=a29fa882a9e3f50c391e3aa08c5c5e72/C, priority=11, startTime=1681527337725; duration=0sec 2023-04-15 02:55:38,597 DEBUG [RS:0;jenkins-hbase20:34681-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-15 02:55:38,598 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a29fa882a9e3f50c391e3aa08c5c5e72, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:55:38,598 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34681] ipc.CallRunner(144): callId: 88 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:40192 deadline: 1681527398596, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a29fa882a9e3f50c391e3aa08c5c5e72, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:38,601 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a29fa882a9e3f50c391e3aa08c5c5e72, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:55:38,601 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34681] ipc.CallRunner(144): callId: 85 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:40234 deadline: 1681527398597, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a29fa882a9e3f50c391e3aa08c5c5e72, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:38,601 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a29fa882a9e3f50c391e3aa08c5c5e72, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:55:38,601 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34681] ipc.CallRunner(144): callId: 87 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:40210 deadline: 1681527398598, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a29fa882a9e3f50c391e3aa08c5c5e72, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:38,647 INFO [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=174 (bloomFilter=true), to=hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/a29fa882a9e3f50c391e3aa08c5c5e72/.tmp/B/d66b99d635ea4fa0adb9e73f822396bf 2023-04-15 02:55:39,057 INFO [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=174 (bloomFilter=true), to=hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/a29fa882a9e3f50c391e3aa08c5c5e72/.tmp/C/7a21071acecd4947a6965346e0be8176 2023-04-15 02:55:39,060 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/a29fa882a9e3f50c391e3aa08c5c5e72/.tmp/A/e968895af35f4ae7a46c8b0e22f49d25 as hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/a29fa882a9e3f50c391e3aa08c5c5e72/A/e968895af35f4ae7a46c8b0e22f49d25 2023-04-15 02:55:39,063 INFO [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/a29fa882a9e3f50c391e3aa08c5c5e72/A/e968895af35f4ae7a46c8b0e22f49d25, entries=150, sequenceid=174, filesize=11.8 K 2023-04-15 02:55:39,063 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/a29fa882a9e3f50c391e3aa08c5c5e72/.tmp/B/d66b99d635ea4fa0adb9e73f822396bf as hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/a29fa882a9e3f50c391e3aa08c5c5e72/B/d66b99d635ea4fa0adb9e73f822396bf 2023-04-15 02:55:39,066 INFO [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/a29fa882a9e3f50c391e3aa08c5c5e72/B/d66b99d635ea4fa0adb9e73f822396bf, entries=150, sequenceid=174, filesize=11.8 K 2023-04-15 02:55:39,066 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/a29fa882a9e3f50c391e3aa08c5c5e72/.tmp/C/7a21071acecd4947a6965346e0be8176 as hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/a29fa882a9e3f50c391e3aa08c5c5e72/C/7a21071acecd4947a6965346e0be8176 2023-04-15 02:55:39,069 INFO [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/a29fa882a9e3f50c391e3aa08c5c5e72/C/7a21071acecd4947a6965346e0be8176, entries=150, sequenceid=174, filesize=11.8 K 2023-04-15 02:55:39,070 INFO [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~114.05 KB/116790, heapSize ~339.38 KB/347520, currentSize=60.38 KB/61830 for a29fa882a9e3f50c391e3aa08c5c5e72 in 1239ms, sequenceid=174, compaction requested=false 2023-04-15 02:55:39,070 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for a29fa882a9e3f50c391e3aa08c5c5e72: 2023-04-15 02:55:39,070 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681527331158.a29fa882a9e3f50c391e3aa08c5c5e72. 2023-04-15 02:55:39,070 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-15 02:55:39,070 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-15 02:55:39,070 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:39,070 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-15 02:55:39,070 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34681,1681527209133' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-15 02:55:39,071 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:39,071 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:39,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:39,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:55:39,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:55:39,071 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:39,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:55:39,071 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-15 02:55:39,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:55:39,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:55:39,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:39,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:55:39,072 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34681,1681527209133' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-15 02:55:39,072 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@14cf59c7[Count = 0] remaining members to acquire global barrier 2023-04-15 02:55:39,072 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-15 02:55:39,072 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:39,073 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:39,073 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:39,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:39,073 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-15 02:55:39,073 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-15 02:55:39,073 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34681,1681527209133' in zk 2023-04-15 02:55:39,073 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:39,073 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-15 02:55:39,074 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:39,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:39,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:55:39,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:55:39,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:55:39,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:55:39,074 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-15 02:55:39,074 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-15 02:55:39,075 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-15 02:55:39,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:55:39,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:39,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:55:39,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:55:39,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:39,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34681,1681527209133': 2023-04-15 02:55:39,076 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34681,1681527209133' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-15 02:55:39,076 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-15 02:55:39,076 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-15 02:55:39,076 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-15 02:55:39,076 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-15 02:55:39,076 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-15 02:55:39,076 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:39,076 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:39,077 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:39,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:39,077 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-15 02:55:39,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:39,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:55:39,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:55:39,077 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-15 02:55:39,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-15 02:55:39,077 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:39,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:55:39,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:39,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:55:39,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:55:39,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:55:39,078 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:39,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:39,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:55:39,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:55:39,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:39,080 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:39,080 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-15 02:55:39,080 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-15 02:55:39,080 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:55:39,080 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-15 02:55:39,080 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-15 02:55:39,080 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-15 02:55:39,080 DEBUG [Thread-2739] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-15 02:55:39,080 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-15 02:55:39,080 DEBUG [Thread-2739] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-15 02:55:39,080 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-15 02:55:39,080 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:55:39,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-15 02:55:39,081 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:39,081 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:39,081 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:39,081 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:39,081 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-15 02:55:39,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-15 02:55:39,105 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34681] regionserver.HRegion(9158): Flush requested on a29fa882a9e3f50c391e3aa08c5c5e72 2023-04-15 02:55:39,105 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing a29fa882a9e3f50c391e3aa08c5c5e72 3/3 column families, dataSize=67.09 KB heapSize=199.97 KB 2023-04-15 02:55:39,106 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a29fa882a9e3f50c391e3aa08c5c5e72, store=A 2023-04-15 02:55:39,106 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-15 02:55:39,106 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a29fa882a9e3f50c391e3aa08c5c5e72, store=B 2023-04-15 02:55:39,106 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-15 02:55:39,106 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a29fa882a9e3f50c391e3aa08c5c5e72, store=C 2023-04-15 02:55:39,106 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-15 02:55:39,139 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a29fa882a9e3f50c391e3aa08c5c5e72, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:55:39,139 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34681] ipc.CallRunner(144): callId: 92 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:40234 deadline: 1681527399133, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a29fa882a9e3f50c391e3aa08c5c5e72, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:39,139 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a29fa882a9e3f50c391e3aa08c5c5e72, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:55:39,139 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34681] ipc.CallRunner(144): callId: 95 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:40210 deadline: 1681527399134, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a29fa882a9e3f50c391e3aa08c5c5e72, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:39,139 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a29fa882a9e3f50c391e3aa08c5c5e72, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:55:39,139 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34681] ipc.CallRunner(144): callId: 96 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:40192 deadline: 1681527399135, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a29fa882a9e3f50c391e3aa08c5c5e72, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:39,181 DEBUG [Thread-2739] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-15 02:55:39,181 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-15 02:55:39,182 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-15 02:55:39,183 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-15 02:55:39,183 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-15 02:55:39,183 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-15 02:55:39,183 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-15 02:55:39,183 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-15 02:55:39,183 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-15 02:55:39,183 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:39,183 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:55:39,184 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-15 02:55:39,184 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:39,184 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-15 02:55:39,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-15 02:55:39,184 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:39,184 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-15 02:55:39,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:55:39,185 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:39,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-15 02:55:39,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:55:39,185 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-15 02:55:39,185 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-15 02:55:39,185 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-15 02:55:39,185 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-15 02:55:39,185 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-15 02:55:39,185 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-15 02:55:39,185 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-15 02:55:39,186 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-15 02:55:39,186 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681527331158.a29fa882a9e3f50c391e3aa08c5c5e72. 2023-04-15 02:55:39,186 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681527331158.a29fa882a9e3f50c391e3aa08c5c5e72. started... 2023-04-15 02:55:39,186 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681527331158.a29fa882a9e3f50c391e3aa08c5c5e72. as already flushing 2023-04-15 02:55:39,186 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681527331158.a29fa882a9e3f50c391e3aa08c5c5e72. 2023-04-15 02:55:39,186 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-15 02:55:39,186 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-15 02:55:39,186 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:39,186 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-15 02:55:39,186 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34681,1681527209133' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-15 02:55:39,187 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:39,187 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:39,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:39,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:55:39,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:55:39,187 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:39,187 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-15 02:55:39,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:55:39,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:55:39,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:55:39,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:39,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:55:39,188 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34681,1681527209133' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-15 02:55:39,188 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@232a7751[Count = 0] remaining members to acquire global barrier 2023-04-15 02:55:39,188 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-15 02:55:39,188 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:39,188 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:39,188 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:39,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:39,188 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-15 02:55:39,188 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-15 02:55:39,188 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34681,1681527209133' in zk 2023-04-15 02:55:39,188 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:39,189 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-15 02:55:39,189 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:39,189 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-15 02:55:39,189 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-15 02:55:39,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:39,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:55:39,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:55:39,189 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-15 02:55:39,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:55:39,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:55:39,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:55:39,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:39,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:55:39,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:55:39,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:39,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34681,1681527209133': 2023-04-15 02:55:39,191 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34681,1681527209133' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-15 02:55:39,191 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-15 02:55:39,191 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-15 02:55:39,191 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-15 02:55:39,191 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-15 02:55:39,191 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-15 02:55:39,192 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:39,192 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:39,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:39,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:55:39,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:55:39,193 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-15 02:55:39,193 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:39,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:39,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:55:39,193 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-15 02:55:39,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-15 02:55:39,193 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:39,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:55:39,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:39,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:55:39,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:55:39,194 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:39,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:39,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:55:39,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:55:39,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:39,206 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-15 02:55:39,206 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-15 02:55:39,206 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:39,206 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-15 02:55:39,206 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-15 02:55:39,206 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:55:39,206 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-15 02:55:39,206 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-15 02:55:39,206 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-15 02:55:39,206 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:55:39,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-15 02:55:39,206 DEBUG [Thread-2739] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-15 02:55:39,206 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:39,206 DEBUG [Thread-2739] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-15 02:55:39,206 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:39,206 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-15 02:55:39,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-15 02:55:39,206 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:39,207 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:39,243 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a29fa882a9e3f50c391e3aa08c5c5e72, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:55:39,243 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34681] ipc.CallRunner(144): callId: 97 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:40210 deadline: 1681527399240, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a29fa882a9e3f50c391e3aa08c5c5e72, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:39,243 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a29fa882a9e3f50c391e3aa08c5c5e72, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:55:39,243 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34681] ipc.CallRunner(144): callId: 94 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:40234 deadline: 1681527399240, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a29fa882a9e3f50c391e3aa08c5c5e72, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:39,243 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a29fa882a9e3f50c391e3aa08c5c5e72, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:55:39,243 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34681] ipc.CallRunner(144): callId: 98 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:40192 deadline: 1681527399240, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a29fa882a9e3f50c391e3aa08c5c5e72, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:39,306 DEBUG [Thread-2739] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-15 02:55:39,307 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-15 02:55:39,308 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-15 02:55:39,309 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-15 02:55:39,309 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-15 02:55:39,309 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-15 02:55:39,309 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-15 02:55:39,309 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-15 02:55:39,309 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-15 02:55:39,309 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:39,309 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:55:39,310 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-15 02:55:39,310 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:39,310 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-15 02:55:39,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-15 02:55:39,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:55:39,310 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:39,310 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-15 02:55:39,311 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:39,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-15 02:55:39,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:55:39,311 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-15 02:55:39,311 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-15 02:55:39,311 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-15 02:55:39,311 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-15 02:55:39,311 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-15 02:55:39,311 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-15 02:55:39,311 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-15 02:55:39,311 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-15 02:55:39,311 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681527331158.a29fa882a9e3f50c391e3aa08c5c5e72. 2023-04-15 02:55:39,311 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681527331158.a29fa882a9e3f50c391e3aa08c5c5e72. started... 2023-04-15 02:55:39,312 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681527331158.a29fa882a9e3f50c391e3aa08c5c5e72. as already flushing 2023-04-15 02:55:39,312 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681527331158.a29fa882a9e3f50c391e3aa08c5c5e72. 2023-04-15 02:55:39,312 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-15 02:55:39,312 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-15 02:55:39,312 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:39,312 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-15 02:55:39,312 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34681,1681527209133' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-15 02:55:39,312 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:39,312 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:39,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:39,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:55:39,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:55:39,313 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:39,313 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-15 02:55:39,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:55:39,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:55:39,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:55:39,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:39,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:55:39,314 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34681,1681527209133' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-15 02:55:39,314 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-15 02:55:39,314 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@34784495[Count = 0] remaining members to acquire global barrier 2023-04-15 02:55:39,314 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:39,314 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:39,314 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:39,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:39,314 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-15 02:55:39,314 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-15 02:55:39,314 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:39,314 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-15 02:55:39,314 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34681,1681527209133' in zk 2023-04-15 02:55:39,315 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:39,315 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-15 02:55:39,315 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-15 02:55:39,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:39,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:55:39,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:55:39,315 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-15 02:55:39,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:55:39,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:55:39,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:55:39,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:39,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:55:39,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:55:39,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:39,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34681,1681527209133': 2023-04-15 02:55:39,317 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34681,1681527209133' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-15 02:55:39,317 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-15 02:55:39,317 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-15 02:55:39,317 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-15 02:55:39,317 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-15 02:55:39,317 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-15 02:55:39,318 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:39,318 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:39,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:39,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:55:39,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:55:39,318 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:39,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:39,318 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-15 02:55:39,318 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:39,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:55:39,318 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-15 02:55:39,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-15 02:55:39,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:55:39,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:39,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:55:39,319 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:39,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:55:39,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:39,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:55:39,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:55:39,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:39,320 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:39,320 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:55:39,320 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:55:39,320 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:39,320 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:39,320 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:39,320 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:39,320 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-15 02:55:39,320 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-15 02:55:39,320 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-15 02:55:39,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-15 02:55:39,320 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-15 02:55:39,320 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-15 02:55:39,321 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-15 02:55:39,321 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-15 02:55:39,321 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-15 02:55:39,321 DEBUG [Thread-2739] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-15 02:55:39,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-15 02:55:39,321 DEBUG [Thread-2739] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-15 02:55:39,421 DEBUG [Thread-2739] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-15 02:55:39,421 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-15 02:55:39,423 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-15 02:55:39,423 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-15 02:55:39,424 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-15 02:55:39,424 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-15 02:55:39,424 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-15 02:55:39,424 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-15 02:55:39,424 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-15 02:55:39,424 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:39,424 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:55:39,425 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-15 02:55:39,425 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:39,425 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-15 02:55:39,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-15 02:55:39,425 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:39,425 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-15 02:55:39,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:55:39,425 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:39,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-15 02:55:39,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:55:39,426 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-15 02:55:39,427 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-15 02:55:39,427 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-15 02:55:39,427 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-15 02:55:39,427 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-15 02:55:39,427 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-15 02:55:39,427 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-15 02:55:39,427 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-15 02:55:39,427 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681527331158.a29fa882a9e3f50c391e3aa08c5c5e72. 2023-04-15 02:55:39,427 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681527331158.a29fa882a9e3f50c391e3aa08c5c5e72. started... 2023-04-15 02:55:39,427 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681527331158.a29fa882a9e3f50c391e3aa08c5c5e72. as already flushing 2023-04-15 02:55:39,427 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681527331158.a29fa882a9e3f50c391e3aa08c5c5e72. 2023-04-15 02:55:39,427 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-15 02:55:39,428 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-15 02:55:39,428 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:39,428 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-15 02:55:39,428 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34681,1681527209133' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-15 02:55:39,428 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:39,428 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:39,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:39,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:55:39,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:55:39,429 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:39,429 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-15 02:55:39,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:55:39,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:55:39,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:55:39,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:39,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:55:39,430 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34681,1681527209133' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-15 02:55:39,430 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@28600dba[Count = 0] remaining members to acquire global barrier 2023-04-15 02:55:39,430 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-15 02:55:39,430 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:39,431 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:39,431 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:39,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:39,431 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-15 02:55:39,431 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-15 02:55:39,431 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34681,1681527209133' in zk 2023-04-15 02:55:39,432 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:39,432 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-15 02:55:39,432 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-15 02:55:39,432 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-15 02:55:39,432 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-15 02:55:39,433 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:39,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:39,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:55:39,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:55:39,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:55:39,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:55:39,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:55:39,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:39,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:55:39,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:55:39,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:39,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34681,1681527209133': 2023-04-15 02:55:39,435 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34681,1681527209133' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-15 02:55:39,435 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-15 02:55:39,435 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-15 02:55:39,435 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-15 02:55:39,435 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-15 02:55:39,435 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-15 02:55:39,436 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:39,436 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-15 02:55:39,436 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:39,436 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:39,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:39,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:39,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:55:39,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:55:39,437 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-15 02:55:39,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-15 02:55:39,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:55:39,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:39,437 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:39,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:55:39,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:55:39,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:55:39,438 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:39,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:39,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:55:39,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:55:39,440 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:39,440 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:55:39,440 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:55:39,440 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:39,440 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-15 02:55:39,440 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-15 02:55:39,440 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-15 02:55:39,440 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-15 02:55:39,440 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:39,440 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:39,440 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:39,440 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-15 02:55:39,440 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-15 02:55:39,440 DEBUG [Thread-2739] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-15 02:55:39,440 DEBUG [Thread-2739] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-15 02:55:39,440 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-15 02:55:39,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-15 02:55:39,444 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a29fa882a9e3f50c391e3aa08c5c5e72, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:55:39,444 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34681] ipc.CallRunner(144): callId: 100 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:40192 deadline: 1681527399444, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a29fa882a9e3f50c391e3aa08c5c5e72, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:39,445 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-15 02:55:39,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-15 02:55:39,446 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a29fa882a9e3f50c391e3aa08c5c5e72, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:55:39,446 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34681] ipc.CallRunner(144): callId: 99 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:40210 deadline: 1681527399446, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a29fa882a9e3f50c391e3aa08c5c5e72, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:39,446 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a29fa882a9e3f50c391e3aa08c5c5e72, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:55:39,446 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34681] ipc.CallRunner(144): callId: 96 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:40234 deadline: 1681527399446, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a29fa882a9e3f50c391e3aa08c5c5e72, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:39,512 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=191 (bloomFilter=true), to=hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/a29fa882a9e3f50c391e3aa08c5c5e72/.tmp/A/38b4f627839a4d0abd642ffe8d380237 2023-04-15 02:55:39,540 DEBUG [Thread-2739] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-15 02:55:39,541 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-15 02:55:39,542 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-15 02:55:39,543 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-15 02:55:39,543 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-15 02:55:39,543 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-15 02:55:39,543 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-15 02:55:39,543 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-15 02:55:39,543 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-15 02:55:39,543 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:39,543 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:55:39,544 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-15 02:55:39,544 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:39,544 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-15 02:55:39,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-15 02:55:39,544 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:39,544 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-15 02:55:39,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:55:39,544 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:39,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-15 02:55:39,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:55:39,545 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-15 02:55:39,545 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-15 02:55:39,545 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-15 02:55:39,545 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-15 02:55:39,545 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-15 02:55:39,545 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-15 02:55:39,545 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-15 02:55:39,545 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-15 02:55:39,545 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681527331158.a29fa882a9e3f50c391e3aa08c5c5e72. 2023-04-15 02:55:39,545 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681527331158.a29fa882a9e3f50c391e3aa08c5c5e72. started... 2023-04-15 02:55:39,545 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681527331158.a29fa882a9e3f50c391e3aa08c5c5e72. as already flushing 2023-04-15 02:55:39,545 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681527331158.a29fa882a9e3f50c391e3aa08c5c5e72. 2023-04-15 02:55:39,545 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-15 02:55:39,545 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-15 02:55:39,545 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:39,545 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-15 02:55:39,545 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34681,1681527209133' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-15 02:55:39,546 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:39,546 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:39,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:39,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:55:39,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:55:39,546 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:39,546 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-15 02:55:39,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:55:39,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:55:39,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:55:39,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:39,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:55:39,547 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34681,1681527209133' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-15 02:55:39,547 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@468f6810[Count = 0] remaining members to acquire global barrier 2023-04-15 02:55:39,547 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-15 02:55:39,547 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:39,548 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:39,548 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:39,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:39,548 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-15 02:55:39,548 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-15 02:55:39,548 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:39,548 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-15 02:55:39,548 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34681,1681527209133' in zk 2023-04-15 02:55:39,549 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-15 02:55:39,549 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-15 02:55:39,549 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:39,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:39,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:55:39,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:55:39,549 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-15 02:55:39,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:55:39,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:55:39,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:55:39,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:39,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:55:39,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:55:39,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:39,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34681,1681527209133': 2023-04-15 02:55:39,551 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34681,1681527209133' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-15 02:55:39,551 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-15 02:55:39,551 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-15 02:55:39,551 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-15 02:55:39,551 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-15 02:55:39,551 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-15 02:55:39,552 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:39,552 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:39,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:39,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:55:39,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:55:39,552 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:39,552 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-15 02:55:39,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:39,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:55:39,552 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:39,552 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-15 02:55:39,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-15 02:55:39,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:55:39,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:39,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:55:39,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:55:39,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:39,554 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:39,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:55:39,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:55:39,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:39,556 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-15 02:55:39,556 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-15 02:55:39,556 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-15 02:55:39,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-15 02:55:39,557 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-15 02:55:39,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-15 02:55:39,559 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:39,559 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:55:39,559 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-15 02:55:39,559 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:55:39,559 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:39,559 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:39,559 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:39,559 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:39,559 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-15 02:55:39,560 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-15 02:55:39,560 DEBUG [Thread-2739] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-15 02:55:39,560 DEBUG [Thread-2739] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-15 02:55:39,660 DEBUG [Thread-2739] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-15 02:55:39,660 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-15 02:55:39,662 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-15 02:55:39,662 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-15 02:55:39,663 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-15 02:55:39,663 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-15 02:55:39,663 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-15 02:55:39,663 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-15 02:55:39,663 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-15 02:55:39,663 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:39,663 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:55:39,664 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:39,664 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-15 02:55:39,664 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-15 02:55:39,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-15 02:55:39,664 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:39,664 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-15 02:55:39,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:55:39,664 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:39,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-15 02:55:39,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:55:39,665 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-15 02:55:39,665 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-15 02:55:39,665 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-15 02:55:39,665 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-15 02:55:39,665 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-15 02:55:39,665 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-15 02:55:39,665 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-15 02:55:39,665 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-15 02:55:39,665 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681527331158.a29fa882a9e3f50c391e3aa08c5c5e72. 2023-04-15 02:55:39,666 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681527331158.a29fa882a9e3f50c391e3aa08c5c5e72. started... 2023-04-15 02:55:39,666 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681527331158.a29fa882a9e3f50c391e3aa08c5c5e72. as already flushing 2023-04-15 02:55:39,666 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681527331158.a29fa882a9e3f50c391e3aa08c5c5e72. 2023-04-15 02:55:39,666 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-15 02:55:39,666 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-15 02:55:39,666 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:39,666 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-15 02:55:39,666 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34681,1681527209133' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-15 02:55:39,667 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:39,667 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:39,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:39,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:55:39,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:55:39,667 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:39,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:55:39,667 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-15 02:55:39,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:55:39,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:55:39,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:39,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:55:39,668 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34681,1681527209133' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-15 02:55:39,668 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-15 02:55:39,668 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@36ef1f6f[Count = 0] remaining members to acquire global barrier 2023-04-15 02:55:39,668 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:39,669 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:39,669 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:39,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:39,669 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-15 02:55:39,669 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-15 02:55:39,669 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34681,1681527209133' in zk 2023-04-15 02:55:39,669 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:39,669 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-15 02:55:39,669 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:39,669 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-15 02:55:39,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:39,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:55:39,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:55:39,669 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-15 02:55:39,670 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-15 02:55:39,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:55:39,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:55:39,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:55:39,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:39,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:55:39,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:55:39,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:39,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34681,1681527209133': 2023-04-15 02:55:39,671 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34681,1681527209133' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-15 02:55:39,671 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-15 02:55:39,671 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-15 02:55:39,671 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-15 02:55:39,671 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-15 02:55:39,671 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-15 02:55:39,672 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:39,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:39,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:55:39,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:55:39,672 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:39,672 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-15 02:55:39,672 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:39,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:39,672 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-15 02:55:39,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-15 02:55:39,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:39,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:55:39,673 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:39,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:55:39,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:55:39,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:55:39,674 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:39,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:39,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:55:39,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:55:39,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:39,675 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:39,675 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-15 02:55:39,675 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:55:39,675 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-15 02:55:39,675 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-15 02:55:39,675 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-15 02:55:39,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-15 02:55:39,675 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-15 02:55:39,675 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-15 02:55:39,676 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-15 02:55:39,675 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:55:39,676 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-15 02:55:39,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-15 02:55:39,676 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:39,676 DEBUG [Thread-2739] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-15 02:55:39,676 DEBUG [Thread-2739] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-15 02:55:39,676 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:39,676 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:39,676 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:39,748 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a29fa882a9e3f50c391e3aa08c5c5e72, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:55:39,748 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34681] ipc.CallRunner(144): callId: 102 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:40192 deadline: 1681527399745, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a29fa882a9e3f50c391e3aa08c5c5e72, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:39,750 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a29fa882a9e3f50c391e3aa08c5c5e72, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:55:39,750 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34681] ipc.CallRunner(144): callId: 101 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:40210 deadline: 1681527399747, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a29fa882a9e3f50c391e3aa08c5c5e72, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:39,750 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a29fa882a9e3f50c391e3aa08c5c5e72, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:55:39,750 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34681] ipc.CallRunner(144): callId: 98 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:40234 deadline: 1681527399748, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a29fa882a9e3f50c391e3aa08c5c5e72, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:39,776 DEBUG [Thread-2739] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-15 02:55:39,776 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-15 02:55:39,777 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-15 02:55:39,778 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-15 02:55:39,778 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-15 02:55:39,778 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-15 02:55:39,778 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-15 02:55:39,778 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-15 02:55:39,778 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-15 02:55:39,779 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:39,779 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:55:39,779 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:39,779 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-15 02:55:39,779 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-15 02:55:39,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-15 02:55:39,780 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:39,780 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-15 02:55:39,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:55:39,780 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:39,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-15 02:55:39,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:55:39,780 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-15 02:55:39,780 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-15 02:55:39,780 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-15 02:55:39,780 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-15 02:55:39,780 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-15 02:55:39,781 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-15 02:55:39,781 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-15 02:55:39,781 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-15 02:55:39,781 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681527331158.a29fa882a9e3f50c391e3aa08c5c5e72. 2023-04-15 02:55:39,781 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681527331158.a29fa882a9e3f50c391e3aa08c5c5e72. started... 2023-04-15 02:55:39,781 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681527331158.a29fa882a9e3f50c391e3aa08c5c5e72. as already flushing 2023-04-15 02:55:39,781 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681527331158.a29fa882a9e3f50c391e3aa08c5c5e72. 2023-04-15 02:55:39,781 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-15 02:55:39,781 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-15 02:55:39,781 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:39,781 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-15 02:55:39,781 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34681,1681527209133' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-15 02:55:39,782 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:39,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:39,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:55:39,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:55:39,782 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:39,784 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:39,784 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-15 02:55:39,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:55:39,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:55:39,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:55:39,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:39,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:55:39,785 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34681,1681527209133' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-15 02:55:39,786 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@40c5a9cb[Count = 0] remaining members to acquire global barrier 2023-04-15 02:55:39,786 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-15 02:55:39,786 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:39,786 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:39,786 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:39,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:39,786 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-15 02:55:39,786 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-15 02:55:39,786 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34681,1681527209133' in zk 2023-04-15 02:55:39,786 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:39,787 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-15 02:55:39,787 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-15 02:55:39,787 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:39,787 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-15 02:55:39,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:39,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:55:39,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:55:39,787 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-15 02:55:39,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:55:39,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:55:39,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:55:39,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:39,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:55:39,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:55:39,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:39,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34681,1681527209133': 2023-04-15 02:55:39,789 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34681,1681527209133' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-15 02:55:39,789 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-15 02:55:39,790 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-15 02:55:39,790 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-15 02:55:39,790 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-15 02:55:39,790 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-15 02:55:39,790 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:39,790 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:39,790 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:39,790 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-15 02:55:39,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:39,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:39,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:55:39,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:55:39,791 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:39,791 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-15 02:55:39,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-15 02:55:39,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:55:39,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:39,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:55:39,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:55:39,792 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:39,792 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:55:39,792 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:39,792 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:55:39,792 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:55:39,793 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:39,793 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-15 02:55:39,793 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:55:39,793 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:55:39,793 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-15 02:55:39,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-15 02:55:39,793 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-15 02:55:39,793 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:39,793 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-15 02:55:39,793 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-15 02:55:39,793 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-15 02:55:39,793 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-15 02:55:39,793 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:39,793 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:39,793 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-15 02:55:39,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-15 02:55:39,793 DEBUG [Thread-2739] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-15 02:55:39,793 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:39,793 DEBUG [Thread-2739] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-15 02:55:39,894 DEBUG [Thread-2739] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-15 02:55:39,894 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-15 02:55:39,896 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-15 02:55:39,896 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-15 02:55:39,897 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-15 02:55:39,897 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-15 02:55:39,897 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-15 02:55:39,897 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-15 02:55:39,897 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-15 02:55:39,897 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:39,897 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:55:39,898 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-15 02:55:39,898 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:39,898 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-15 02:55:39,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-15 02:55:39,898 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:39,898 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-15 02:55:39,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:55:39,898 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:39,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-15 02:55:39,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:55:39,898 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-15 02:55:39,899 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-15 02:55:39,899 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-15 02:55:39,899 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-15 02:55:39,899 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-15 02:55:39,899 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-15 02:55:39,899 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-15 02:55:39,899 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-15 02:55:39,899 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681527331158.a29fa882a9e3f50c391e3aa08c5c5e72. 2023-04-15 02:55:39,899 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681527331158.a29fa882a9e3f50c391e3aa08c5c5e72. started... 2023-04-15 02:55:39,899 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681527331158.a29fa882a9e3f50c391e3aa08c5c5e72. as already flushing 2023-04-15 02:55:39,899 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681527331158.a29fa882a9e3f50c391e3aa08c5c5e72. 2023-04-15 02:55:39,899 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-15 02:55:39,899 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-15 02:55:39,899 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:39,899 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-15 02:55:39,899 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34681,1681527209133' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-15 02:55:39,900 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:39,900 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:39,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:39,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:55:39,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:55:39,900 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:39,900 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-15 02:55:39,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:55:39,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:55:39,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:55:39,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:39,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:55:39,901 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34681,1681527209133' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-15 02:55:39,901 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@68e071b4[Count = 0] remaining members to acquire global barrier 2023-04-15 02:55:39,901 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-15 02:55:39,901 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:39,901 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:39,902 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:39,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:39,902 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-15 02:55:39,902 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-15 02:55:39,902 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:39,902 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-15 02:55:39,902 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34681,1681527209133' in zk 2023-04-15 02:55:39,902 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:39,902 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-15 02:55:39,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:39,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:55:39,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:55:39,902 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-15 02:55:39,903 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-15 02:55:39,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:55:39,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:55:39,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:55:39,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:39,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:55:39,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:55:39,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:39,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34681,1681527209133': 2023-04-15 02:55:39,904 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34681,1681527209133' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-15 02:55:39,904 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-15 02:55:39,904 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-15 02:55:39,904 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-15 02:55:39,904 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-15 02:55:39,904 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-15 02:55:39,905 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:39,905 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:39,905 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-15 02:55:39,905 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:39,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:39,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:55:39,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:55:39,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:39,905 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:39,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:55:39,905 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-15 02:55:39,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-15 02:55:39,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:55:39,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:39,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:55:39,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:55:39,906 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:39,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:39,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:55:39,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:55:39,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:39,908 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:39,908 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:55:39,908 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-15 02:55:39,908 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:55:39,908 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-15 02:55:39,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-15 02:55:39,908 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-15 02:55:39,908 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-15 02:55:39,908 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:39,908 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:39,908 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:39,908 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:39,908 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-15 02:55:39,908 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-15 02:55:39,908 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-15 02:55:39,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-15 02:55:39,908 DEBUG [Thread-2739] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-15 02:55:39,908 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-15 02:55:39,908 DEBUG [Thread-2739] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-15 02:55:39,920 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=191 (bloomFilter=true), to=hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/a29fa882a9e3f50c391e3aa08c5c5e72/.tmp/B/c6afbb3f63ab49ae98fca6be4fd386a0 2023-04-15 02:55:40,008 DEBUG [Thread-2739] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-15 02:55:40,009 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-15 02:55:40,010 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-15 02:55:40,011 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-15 02:55:40,011 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-15 02:55:40,011 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-15 02:55:40,011 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-15 02:55:40,011 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-15 02:55:40,011 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-15 02:55:40,011 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:40,011 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:55:40,012 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-15 02:55:40,012 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:40,012 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-15 02:55:40,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-15 02:55:40,012 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:40,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:55:40,012 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-15 02:55:40,013 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:40,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-15 02:55:40,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:55:40,013 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-15 02:55:40,013 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-15 02:55:40,013 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-15 02:55:40,013 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-15 02:55:40,013 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-15 02:55:40,013 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-15 02:55:40,013 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-15 02:55:40,013 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-15 02:55:40,013 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681527331158.a29fa882a9e3f50c391e3aa08c5c5e72. 2023-04-15 02:55:40,013 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681527331158.a29fa882a9e3f50c391e3aa08c5c5e72. started... 2023-04-15 02:55:40,014 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681527331158.a29fa882a9e3f50c391e3aa08c5c5e72. as already flushing 2023-04-15 02:55:40,014 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681527331158.a29fa882a9e3f50c391e3aa08c5c5e72. 2023-04-15 02:55:40,014 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-15 02:55:40,014 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-15 02:55:40,014 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:40,014 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-15 02:55:40,014 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34681,1681527209133' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-15 02:55:40,014 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:40,014 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:40,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:40,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:55:40,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:55:40,015 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:40,015 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-15 02:55:40,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:55:40,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:55:40,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:55:40,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:40,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:55:40,016 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34681,1681527209133' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-15 02:55:40,016 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-15 02:55:40,016 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@55925b50[Count = 0] remaining members to acquire global barrier 2023-04-15 02:55:40,016 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:40,016 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:40,016 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:40,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:40,017 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-15 02:55:40,017 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-15 02:55:40,017 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34681,1681527209133' in zk 2023-04-15 02:55:40,017 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:40,017 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-15 02:55:40,017 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-15 02:55:40,017 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:40,017 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-15 02:55:40,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:40,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:55:40,017 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-15 02:55:40,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:55:40,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:55:40,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:55:40,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:55:40,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:40,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:55:40,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:55:40,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:40,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34681,1681527209133': 2023-04-15 02:55:40,019 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34681,1681527209133' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-15 02:55:40,019 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-15 02:55:40,019 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-15 02:55:40,019 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-15 02:55:40,019 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-15 02:55:40,019 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-15 02:55:40,020 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:40,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:40,020 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:40,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:55:40,020 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:40,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:55:40,020 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-15 02:55:40,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:40,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:55:40,021 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-15 02:55:40,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-15 02:55:40,021 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:40,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:55:40,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:40,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:55:40,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:55:40,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:40,022 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:40,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:55:40,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:55:40,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:40,025 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:40,025 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-15 02:55:40,025 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:55:40,025 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-15 02:55:40,025 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-15 02:55:40,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-15 02:55:40,025 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-15 02:55:40,025 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:55:40,025 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-15 02:55:40,025 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-15 02:55:40,025 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-15 02:55:40,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-15 02:55:40,025 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:40,025 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:40,025 DEBUG [Thread-2739] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-15 02:55:40,025 DEBUG [Thread-2739] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-15 02:55:40,025 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:40,026 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:40,126 DEBUG [Thread-2739] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-15 02:55:40,126 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-15 02:55:40,127 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-15 02:55:40,128 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-15 02:55:40,128 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-15 02:55:40,128 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-15 02:55:40,128 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-15 02:55:40,128 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-15 02:55:40,128 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-15 02:55:40,128 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:40,128 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:55:40,129 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-15 02:55:40,129 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:40,129 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-15 02:55:40,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-15 02:55:40,129 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:40,129 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-15 02:55:40,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:55:40,130 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:40,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-15 02:55:40,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:55:40,130 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-15 02:55:40,130 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-15 02:55:40,130 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-15 02:55:40,130 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-15 02:55:40,130 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-15 02:55:40,130 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-15 02:55:40,131 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-15 02:55:40,131 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-15 02:55:40,131 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681527331158.a29fa882a9e3f50c391e3aa08c5c5e72. 2023-04-15 02:55:40,131 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681527331158.a29fa882a9e3f50c391e3aa08c5c5e72. started... 2023-04-15 02:55:40,131 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681527331158.a29fa882a9e3f50c391e3aa08c5c5e72. as already flushing 2023-04-15 02:55:40,131 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681527331158.a29fa882a9e3f50c391e3aa08c5c5e72. 2023-04-15 02:55:40,131 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-15 02:55:40,131 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-15 02:55:40,131 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:40,131 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-15 02:55:40,131 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34681,1681527209133' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-15 02:55:40,132 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:40,132 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:40,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:40,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:55:40,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:55:40,132 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:40,132 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-15 02:55:40,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:55:40,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:55:40,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:55:40,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:40,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:55:40,133 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34681,1681527209133' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-15 02:55:40,133 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-15 02:55:40,133 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1bc832ea[Count = 0] remaining members to acquire global barrier 2023-04-15 02:55:40,133 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:40,134 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:40,134 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:40,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:40,134 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-15 02:55:40,134 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-15 02:55:40,134 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:40,134 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-15 02:55:40,134 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34681,1681527209133' in zk 2023-04-15 02:55:40,134 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-15 02:55:40,135 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-15 02:55:40,135 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-15 02:55:40,135 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:40,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:40,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:55:40,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:55:40,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:55:40,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:55:40,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:55:40,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:40,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:55:40,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:55:40,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:40,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34681,1681527209133': 2023-04-15 02:55:40,136 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34681,1681527209133' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-15 02:55:40,136 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-15 02:55:40,137 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-15 02:55:40,137 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-15 02:55:40,137 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-15 02:55:40,137 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-15 02:55:40,137 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:40,137 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:40,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:40,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:55:40,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:55:40,137 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:40,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:40,137 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-15 02:55:40,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:55:40,138 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-15 02:55:40,138 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:40,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:55:40,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-15 02:55:40,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:40,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:55:40,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:55:40,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:40,139 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:40,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:55:40,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:55:40,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:40,140 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:40,140 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-15 02:55:40,140 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:55:40,140 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-15 02:55:40,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-15 02:55:40,140 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-15 02:55:40,140 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-15 02:55:40,140 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-15 02:55:40,140 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-15 02:55:40,140 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:55:40,140 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:40,140 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:40,140 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:40,140 DEBUG [Thread-2739] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-15 02:55:40,140 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-15 02:55:40,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-15 02:55:40,140 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:40,140 DEBUG [Thread-2739] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-15 02:55:40,193 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a29fa882a9e3f50c391e3aa08c5c5e72, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:55:40,193 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34681] ipc.CallRunner(144): callId: 67 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:40158 deadline: 1681527400190, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a29fa882a9e3f50c391e3aa08c5c5e72, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:40,195 DEBUG [Thread-2731] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4167 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a29fa882a9e3f50c391e3aa08c5c5e72, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_1' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1681527331158.a29fa882a9e3f50c391e3aa08c5c5e72., hostname=jenkins-hbase20.apache.org,34681,1681527209133, seqNum=2, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a29fa882a9e3f50c391e3aa08c5c5e72, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor31.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a29fa882a9e3f50c391e3aa08c5c5e72, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-15 02:55:40,201 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a29fa882a9e3f50c391e3aa08c5c5e72, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:55:40,201 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34681] ipc.CallRunner(144): callId: 68 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:40212 deadline: 1681527400199, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a29fa882a9e3f50c391e3aa08c5c5e72, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:40,202 DEBUG [Thread-2733] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4175 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a29fa882a9e3f50c391e3aa08c5c5e72, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_2' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1681527331158.a29fa882a9e3f50c391e3aa08c5c5e72., hostname=jenkins-hbase20.apache.org,34681,1681527209133, seqNum=2, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a29fa882a9e3f50c391e3aa08c5c5e72, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor31.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a29fa882a9e3f50c391e3aa08c5c5e72, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-15 02:55:40,241 DEBUG [Thread-2739] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-15 02:55:40,241 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-15 02:55:40,242 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-15 02:55:40,243 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-15 02:55:40,243 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-15 02:55:40,243 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-15 02:55:40,243 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-15 02:55:40,243 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-15 02:55:40,243 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-15 02:55:40,243 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:40,243 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:55:40,244 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:40,244 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-15 02:55:40,244 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-15 02:55:40,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-15 02:55:40,244 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:40,244 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-15 02:55:40,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:55:40,244 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:40,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-15 02:55:40,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:55:40,245 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-15 02:55:40,245 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-15 02:55:40,245 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-15 02:55:40,245 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-15 02:55:40,245 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-15 02:55:40,245 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-15 02:55:40,245 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-15 02:55:40,245 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-15 02:55:40,245 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681527331158.a29fa882a9e3f50c391e3aa08c5c5e72. 2023-04-15 02:55:40,245 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681527331158.a29fa882a9e3f50c391e3aa08c5c5e72. started... 2023-04-15 02:55:40,245 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681527331158.a29fa882a9e3f50c391e3aa08c5c5e72. as already flushing 2023-04-15 02:55:40,245 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681527331158.a29fa882a9e3f50c391e3aa08c5c5e72. 2023-04-15 02:55:40,245 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-15 02:55:40,245 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-15 02:55:40,245 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:40,245 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-15 02:55:40,245 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34681,1681527209133' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-15 02:55:40,247 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:40,247 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:40,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:40,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:55:40,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:55:40,247 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:40,247 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-15 02:55:40,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:55:40,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:55:40,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:55:40,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:40,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:55:40,248 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34681,1681527209133' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-15 02:55:40,248 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@762e5ba9[Count = 0] remaining members to acquire global barrier 2023-04-15 02:55:40,248 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-15 02:55:40,248 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:40,249 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:40,249 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:40,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:40,249 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-15 02:55:40,249 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-15 02:55:40,249 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:40,249 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-15 02:55:40,249 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34681,1681527209133' in zk 2023-04-15 02:55:40,250 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:40,250 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-15 02:55:40,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:40,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:55:40,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:55:40,250 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-15 02:55:40,250 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-15 02:55:40,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:55:40,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:55:40,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:55:40,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:40,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:55:40,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:55:40,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:40,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34681,1681527209133': 2023-04-15 02:55:40,252 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34681,1681527209133' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-15 02:55:40,252 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-15 02:55:40,252 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-15 02:55:40,252 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-15 02:55:40,252 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-15 02:55:40,252 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-15 02:55:40,252 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:40,252 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:40,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:40,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:55:40,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:55:40,253 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:40,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:40,253 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-15 02:55:40,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:55:40,253 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:40,253 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-15 02:55:40,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-15 02:55:40,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:55:40,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:40,253 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a29fa882a9e3f50c391e3aa08c5c5e72, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:55:40,253 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34681] ipc.CallRunner(144): callId: 104 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:40192 deadline: 1681527400249, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a29fa882a9e3f50c391e3aa08c5c5e72, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:40,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:55:40,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:55:40,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:40,254 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:40,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:55:40,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:55:40,255 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a29fa882a9e3f50c391e3aa08c5c5e72, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:55:40,255 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34681] ipc.CallRunner(144): callId: 100 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:40234 deadline: 1681527400251, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a29fa882a9e3f50c391e3aa08c5c5e72, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:40,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:40,256 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:40,256 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-15 02:55:40,256 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:55:40,256 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-15 02:55:40,256 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-15 02:55:40,256 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-15 02:55:40,256 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-15 02:55:40,256 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-15 02:55:40,256 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:55:40,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-15 02:55:40,256 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:40,256 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:40,256 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:40,256 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:40,256 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-15 02:55:40,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-15 02:55:40,256 DEBUG [Thread-2739] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-15 02:55:40,256 DEBUG [Thread-2739] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-15 02:55:40,256 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a29fa882a9e3f50c391e3aa08c5c5e72, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:55:40,257 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34681] ipc.CallRunner(144): callId: 103 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:40210 deadline: 1681527400254, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a29fa882a9e3f50c391e3aa08c5c5e72, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:40,328 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=191 (bloomFilter=true), to=hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/a29fa882a9e3f50c391e3aa08c5c5e72/.tmp/C/9499e2ae026b4cf7933d82cabe340d82 2023-04-15 02:55:40,331 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/a29fa882a9e3f50c391e3aa08c5c5e72/.tmp/A/38b4f627839a4d0abd642ffe8d380237 as hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/a29fa882a9e3f50c391e3aa08c5c5e72/A/38b4f627839a4d0abd642ffe8d380237 2023-04-15 02:55:40,333 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/a29fa882a9e3f50c391e3aa08c5c5e72/A/38b4f627839a4d0abd642ffe8d380237, entries=250, sequenceid=191, filesize=16.4 K 2023-04-15 02:55:40,334 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/a29fa882a9e3f50c391e3aa08c5c5e72/.tmp/B/c6afbb3f63ab49ae98fca6be4fd386a0 as hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/a29fa882a9e3f50c391e3aa08c5c5e72/B/c6afbb3f63ab49ae98fca6be4fd386a0 2023-04-15 02:55:40,337 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/a29fa882a9e3f50c391e3aa08c5c5e72/B/c6afbb3f63ab49ae98fca6be4fd386a0, entries=150, sequenceid=191, filesize=11.8 K 2023-04-15 02:55:40,337 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/a29fa882a9e3f50c391e3aa08c5c5e72/.tmp/C/9499e2ae026b4cf7933d82cabe340d82 as hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/a29fa882a9e3f50c391e3aa08c5c5e72/C/9499e2ae026b4cf7933d82cabe340d82 2023-04-15 02:55:40,340 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/a29fa882a9e3f50c391e3aa08c5c5e72/C/9499e2ae026b4cf7933d82cabe340d82, entries=150, sequenceid=191, filesize=11.8 K 2023-04-15 02:55:40,340 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~73.80 KB/75570, heapSize ~219.84 KB/225120, currentSize=100.63 KB/103050 for a29fa882a9e3f50c391e3aa08c5c5e72 in 1235ms, sequenceid=191, compaction requested=true 2023-04-15 02:55:40,340 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for a29fa882a9e3f50c391e3aa08c5c5e72: 2023-04-15 02:55:40,340 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-15 02:55:40,340 DEBUG [RS:0;jenkins-hbase20:34681-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-15 02:55:40,340 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-15 02:55:40,340 DEBUG [RS:0;jenkins-hbase20:34681-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-15 02:55:40,341 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-15 02:55:40,341 DEBUG [RS:0;jenkins-hbase20:34681-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 41162 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-15 02:55:40,341 DEBUG [RS:0;jenkins-hbase20:34681-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36382 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-15 02:55:40,341 DEBUG [RS:0;jenkins-hbase20:34681-shortCompactions-0] regionserver.HStore(1912): a29fa882a9e3f50c391e3aa08c5c5e72/A is initiating minor compaction (all files) 2023-04-15 02:55:40,341 DEBUG [RS:0;jenkins-hbase20:34681-longCompactions-0] regionserver.HStore(1912): a29fa882a9e3f50c391e3aa08c5c5e72/B is initiating minor compaction (all files) 2023-04-15 02:55:40,341 INFO [RS:0;jenkins-hbase20:34681-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of a29fa882a9e3f50c391e3aa08c5c5e72/A in TestAcidGuarantees,,1681527331158.a29fa882a9e3f50c391e3aa08c5c5e72. 2023-04-15 02:55:40,341 INFO [RS:0;jenkins-hbase20:34681-longCompactions-0] regionserver.HRegion(2259): Starting compaction of a29fa882a9e3f50c391e3aa08c5c5e72/B in TestAcidGuarantees,,1681527331158.a29fa882a9e3f50c391e3aa08c5c5e72. 2023-04-15 02:55:40,341 INFO [RS:0;jenkins-hbase20:34681-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/a29fa882a9e3f50c391e3aa08c5c5e72/A/566f729b22db4659ab9e6ff55d4bfc41, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/a29fa882a9e3f50c391e3aa08c5c5e72/A/e968895af35f4ae7a46c8b0e22f49d25, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/a29fa882a9e3f50c391e3aa08c5c5e72/A/38b4f627839a4d0abd642ffe8d380237] into tmpdir=hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/a29fa882a9e3f50c391e3aa08c5c5e72/.tmp, totalSize=40.2 K 2023-04-15 02:55:40,341 INFO [RS:0;jenkins-hbase20:34681-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/a29fa882a9e3f50c391e3aa08c5c5e72/B/8669bced50064421bc35c2bbf8ba2f27, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/a29fa882a9e3f50c391e3aa08c5c5e72/B/d66b99d635ea4fa0adb9e73f822396bf, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/a29fa882a9e3f50c391e3aa08c5c5e72/B/c6afbb3f63ab49ae98fca6be4fd386a0] into tmpdir=hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/a29fa882a9e3f50c391e3aa08c5c5e72/.tmp, totalSize=35.5 K 2023-04-15 02:55:40,342 DEBUG [RS:0;jenkins-hbase20:34681-shortCompactions-0] compactions.Compactor(207): Compacting 566f729b22db4659ab9e6ff55d4bfc41, keycount=150, bloomtype=ROW, size=12.0 K, encoding=NONE, compression=NONE, seqNum=154, earliestPutTs=1681527337148 2023-04-15 02:55:40,342 DEBUG [RS:0;jenkins-hbase20:34681-longCompactions-0] compactions.Compactor(207): Compacting 8669bced50064421bc35c2bbf8ba2f27, keycount=150, bloomtype=ROW, size=12.0 K, encoding=NONE, compression=NONE, seqNum=154, earliestPutTs=1681527337148 2023-04-15 02:55:40,342 DEBUG [RS:0;jenkins-hbase20:34681-shortCompactions-0] compactions.Compactor(207): Compacting e968895af35f4ae7a46c8b0e22f49d25, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=174, earliestPutTs=1681527337304 2023-04-15 02:55:40,342 DEBUG [RS:0;jenkins-hbase20:34681-longCompactions-0] compactions.Compactor(207): Compacting d66b99d635ea4fa0adb9e73f822396bf, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=174, earliestPutTs=1681527337304 2023-04-15 02:55:40,342 DEBUG [RS:0;jenkins-hbase20:34681-shortCompactions-0] compactions.Compactor(207): Compacting 38b4f627839a4d0abd642ffe8d380237, keycount=250, bloomtype=ROW, size=16.4 K, encoding=NONE, compression=NONE, seqNum=191, earliestPutTs=1681527337939 2023-04-15 02:55:40,342 DEBUG [RS:0;jenkins-hbase20:34681-longCompactions-0] compactions.Compactor(207): Compacting c6afbb3f63ab49ae98fca6be4fd386a0, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=191, earliestPutTs=1681527337970 2023-04-15 02:55:40,346 INFO [RS:0;jenkins-hbase20:34681-shortCompactions-0] throttle.PressureAwareThroughputController(145): a29fa882a9e3f50c391e3aa08c5c5e72#A#compaction#377 average throughput is unlimited, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-15 02:55:40,348 INFO [RS:0;jenkins-hbase20:34681-longCompactions-0] throttle.PressureAwareThroughputController(145): a29fa882a9e3f50c391e3aa08c5c5e72#B#compaction#378 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-15 02:55:40,356 DEBUG [RS:0;jenkins-hbase20:34681-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/a29fa882a9e3f50c391e3aa08c5c5e72/.tmp/A/4ab24298f79e488abab5ebc46292e6b3 as hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/a29fa882a9e3f50c391e3aa08c5c5e72/A/4ab24298f79e488abab5ebc46292e6b3 2023-04-15 02:55:40,356 DEBUG [Thread-2739] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-15 02:55:40,357 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-15 02:55:40,358 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-15 02:55:40,359 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-15 02:55:40,359 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-15 02:55:40,359 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-15 02:55:40,359 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-15 02:55:40,359 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-15 02:55:40,359 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-15 02:55:40,359 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:40,359 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:55:40,360 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-15 02:55:40,360 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:40,360 INFO [RS:0;jenkins-hbase20:34681-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in a29fa882a9e3f50c391e3aa08c5c5e72/A of a29fa882a9e3f50c391e3aa08c5c5e72 into 4ab24298f79e488abab5ebc46292e6b3(size=12.1 K), total size for store is 12.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-15 02:55:40,360 DEBUG [RS:0;jenkins-hbase20:34681-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for a29fa882a9e3f50c391e3aa08c5c5e72: 2023-04-15 02:55:40,360 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-15 02:55:40,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-15 02:55:40,360 INFO [RS:0;jenkins-hbase20:34681-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681527331158.a29fa882a9e3f50c391e3aa08c5c5e72., storeName=a29fa882a9e3f50c391e3aa08c5c5e72/A, priority=13, startTime=1681527340340; duration=0sec 2023-04-15 02:55:40,361 DEBUG [RS:0;jenkins-hbase20:34681-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-15 02:55:40,361 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:40,361 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-15 02:55:40,361 DEBUG [RS:0;jenkins-hbase20:34681-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-15 02:55:40,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:55:40,361 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:40,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-15 02:55:40,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:55:40,361 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-15 02:55:40,362 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-15 02:55:40,362 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-15 02:55:40,362 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-15 02:55:40,362 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-15 02:55:40,362 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-15 02:55:40,362 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-15 02:55:40,362 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-15 02:55:40,362 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681527331158.a29fa882a9e3f50c391e3aa08c5c5e72. 2023-04-15 02:55:40,362 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681527331158.a29fa882a9e3f50c391e3aa08c5c5e72. started... 2023-04-15 02:55:40,362 INFO [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing a29fa882a9e3f50c391e3aa08c5c5e72 3/3 column families, dataSize=100.63 KB heapSize=299.58 KB 2023-04-15 02:55:40,362 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a29fa882a9e3f50c391e3aa08c5c5e72, store=A 2023-04-15 02:55:40,363 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-15 02:55:40,363 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a29fa882a9e3f50c391e3aa08c5c5e72, store=B 2023-04-15 02:55:40,363 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-15 02:55:40,363 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a29fa882a9e3f50c391e3aa08c5c5e72, store=C 2023-04-15 02:55:40,363 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-15 02:55:40,362 DEBUG [RS:0;jenkins-hbase20:34681-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36382 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-15 02:55:40,363 DEBUG [RS:0;jenkins-hbase20:34681-shortCompactions-0] regionserver.HStore(1912): a29fa882a9e3f50c391e3aa08c5c5e72/C is initiating minor compaction (all files) 2023-04-15 02:55:40,363 INFO [RS:0;jenkins-hbase20:34681-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of a29fa882a9e3f50c391e3aa08c5c5e72/C in TestAcidGuarantees,,1681527331158.a29fa882a9e3f50c391e3aa08c5c5e72. 2023-04-15 02:55:40,363 INFO [RS:0;jenkins-hbase20:34681-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/a29fa882a9e3f50c391e3aa08c5c5e72/C/7fdf48a747e543e1bc9b832166cde47a, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/a29fa882a9e3f50c391e3aa08c5c5e72/C/7a21071acecd4947a6965346e0be8176, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/a29fa882a9e3f50c391e3aa08c5c5e72/C/9499e2ae026b4cf7933d82cabe340d82] into tmpdir=hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/a29fa882a9e3f50c391e3aa08c5c5e72/.tmp, totalSize=35.5 K 2023-04-15 02:55:40,364 DEBUG [RS:0;jenkins-hbase20:34681-shortCompactions-0] compactions.Compactor(207): Compacting 7fdf48a747e543e1bc9b832166cde47a, keycount=150, bloomtype=ROW, size=12.0 K, encoding=NONE, compression=NONE, seqNum=154, earliestPutTs=1681527337148 2023-04-15 02:55:40,364 DEBUG [RS:0;jenkins-hbase20:34681-shortCompactions-0] compactions.Compactor(207): Compacting 7a21071acecd4947a6965346e0be8176, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=174, earliestPutTs=1681527337304 2023-04-15 02:55:40,364 DEBUG [RS:0;jenkins-hbase20:34681-shortCompactions-0] compactions.Compactor(207): Compacting 9499e2ae026b4cf7933d82cabe340d82, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=191, earliestPutTs=1681527337970 2023-04-15 02:55:40,366 DEBUG [RS:0;jenkins-hbase20:34681-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/a29fa882a9e3f50c391e3aa08c5c5e72/.tmp/B/af8ce0b6d48546668e62dac87f2fbbb9 as hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/a29fa882a9e3f50c391e3aa08c5c5e72/B/af8ce0b6d48546668e62dac87f2fbbb9 2023-04-15 02:55:40,368 INFO [RS:0;jenkins-hbase20:34681-shortCompactions-0] throttle.PressureAwareThroughputController(145): a29fa882a9e3f50c391e3aa08c5c5e72#C#compaction#380 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-15 02:55:40,370 INFO [RS:0;jenkins-hbase20:34681-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in a29fa882a9e3f50c391e3aa08c5c5e72/B of a29fa882a9e3f50c391e3aa08c5c5e72 into af8ce0b6d48546668e62dac87f2fbbb9(size=12.1 K), total size for store is 12.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-15 02:55:40,371 DEBUG [RS:0;jenkins-hbase20:34681-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for a29fa882a9e3f50c391e3aa08c5c5e72: 2023-04-15 02:55:40,371 INFO [RS:0;jenkins-hbase20:34681-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681527331158.a29fa882a9e3f50c391e3aa08c5c5e72., storeName=a29fa882a9e3f50c391e3aa08c5c5e72/B, priority=13, startTime=1681527340340; duration=0sec 2023-04-15 02:55:40,371 DEBUG [RS:0;jenkins-hbase20:34681-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-15 02:55:40,374 INFO [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=210 (bloomFilter=true), to=hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/a29fa882a9e3f50c391e3aa08c5c5e72/.tmp/A/d2705aa6585d42dba39e3bbfd4abf7ef 2023-04-15 02:55:40,378 DEBUG [RS:0;jenkins-hbase20:34681-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/a29fa882a9e3f50c391e3aa08c5c5e72/.tmp/C/ad509ed0bc7e4ae5b4d80a55856ce60f as hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/a29fa882a9e3f50c391e3aa08c5c5e72/C/ad509ed0bc7e4ae5b4d80a55856ce60f 2023-04-15 02:55:40,381 INFO [RS:0;jenkins-hbase20:34681-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in a29fa882a9e3f50c391e3aa08c5c5e72/C of a29fa882a9e3f50c391e3aa08c5c5e72 into ad509ed0bc7e4ae5b4d80a55856ce60f(size=12.1 K), total size for store is 12.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-15 02:55:40,381 DEBUG [RS:0;jenkins-hbase20:34681-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for a29fa882a9e3f50c391e3aa08c5c5e72: 2023-04-15 02:55:40,381 INFO [RS:0;jenkins-hbase20:34681-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681527331158.a29fa882a9e3f50c391e3aa08c5c5e72., storeName=a29fa882a9e3f50c391e3aa08c5c5e72/C, priority=13, startTime=1681527340341; duration=0sec 2023-04-15 02:55:40,381 DEBUG [RS:0;jenkins-hbase20:34681-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-15 02:55:40,786 INFO [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=210 (bloomFilter=true), to=hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/a29fa882a9e3f50c391e3aa08c5c5e72/.tmp/B/28da064938ee4d62a7003f2600f11230 2023-04-15 02:55:41,205 INFO [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=210 (bloomFilter=true), to=hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/a29fa882a9e3f50c391e3aa08c5c5e72/.tmp/C/8707dafd31ff48eb9c84d4e178e25eda 2023-04-15 02:55:41,211 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/a29fa882a9e3f50c391e3aa08c5c5e72/.tmp/A/d2705aa6585d42dba39e3bbfd4abf7ef as hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/a29fa882a9e3f50c391e3aa08c5c5e72/A/d2705aa6585d42dba39e3bbfd4abf7ef 2023-04-15 02:55:41,215 INFO [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/a29fa882a9e3f50c391e3aa08c5c5e72/A/d2705aa6585d42dba39e3bbfd4abf7ef, entries=150, sequenceid=210, filesize=11.8 K 2023-04-15 02:55:41,216 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/a29fa882a9e3f50c391e3aa08c5c5e72/.tmp/B/28da064938ee4d62a7003f2600f11230 as hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/a29fa882a9e3f50c391e3aa08c5c5e72/B/28da064938ee4d62a7003f2600f11230 2023-04-15 02:55:41,224 INFO [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/a29fa882a9e3f50c391e3aa08c5c5e72/B/28da064938ee4d62a7003f2600f11230, entries=150, sequenceid=210, filesize=11.8 K 2023-04-15 02:55:41,224 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/a29fa882a9e3f50c391e3aa08c5c5e72/.tmp/C/8707dafd31ff48eb9c84d4e178e25eda as hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/a29fa882a9e3f50c391e3aa08c5c5e72/C/8707dafd31ff48eb9c84d4e178e25eda 2023-04-15 02:55:41,231 INFO [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/a29fa882a9e3f50c391e3aa08c5c5e72/C/8707dafd31ff48eb9c84d4e178e25eda, entries=150, sequenceid=210, filesize=11.8 K 2023-04-15 02:55:41,232 INFO [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~100.63 KB/103050, heapSize ~299.53 KB/306720, currentSize=0 B/0 for a29fa882a9e3f50c391e3aa08c5c5e72 in 870ms, sequenceid=210, compaction requested=false 2023-04-15 02:55:41,232 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for a29fa882a9e3f50c391e3aa08c5c5e72: 2023-04-15 02:55:41,232 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681527331158.a29fa882a9e3f50c391e3aa08c5c5e72. 2023-04-15 02:55:41,232 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-15 02:55:41,232 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-15 02:55:41,232 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:41,234 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-15 02:55:41,234 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34681,1681527209133' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-15 02:55:41,235 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:41,235 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:41,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:41,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:55:41,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:55:41,236 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:41,236 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-15 02:55:41,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:55:41,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:55:41,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:55:41,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:41,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:55:41,238 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34681,1681527209133' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-15 02:55:41,238 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1d28f909[Count = 0] remaining members to acquire global barrier 2023-04-15 02:55:41,238 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-15 02:55:41,238 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:41,239 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:41,239 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:41,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:41,239 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-15 02:55:41,239 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-15 02:55:41,239 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34681,1681527209133' in zk 2023-04-15 02:55:41,239 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:41,239 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-15 02:55:41,240 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:41,240 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-15 02:55:41,240 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-15 02:55:41,240 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-15 02:55:41,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:41,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:55:41,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:55:41,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:55:41,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:55:41,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:55:41,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:41,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:55:41,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:55:41,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:41,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34681,1681527209133': 2023-04-15 02:55:41,242 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34681,1681527209133' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-15 02:55:41,242 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-15 02:55:41,242 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-15 02:55:41,243 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-15 02:55:41,243 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-15 02:55:41,243 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-15 02:55:41,244 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:41,244 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-15 02:55:41,244 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:41,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:41,244 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:41,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:41,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:55:41,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:55:41,244 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-15 02:55:41,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-15 02:55:41,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:55:41,245 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:41,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:41,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:55:41,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:55:41,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:55:41,245 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:41,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:41,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:55:41,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:55:41,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:41,247 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:41,247 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:55:41,247 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:55:41,247 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-15 02:55:41,247 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-15 02:55:41,247 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-15 02:55:41,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-15 02:55:41,247 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-15 02:55:41,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-15 02:55:41,247 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-15 02:55:41,248 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-15 02:55:41,247 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:41,248 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-15 02:55:41,247 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-15 02:55:41,248 DEBUG [Thread-2739] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-15 02:55:41,248 DEBUG [Thread-2739] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-15 02:55:41,248 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:41,248 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:41,248 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:41,264 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34681] regionserver.HRegion(9158): Flush requested on a29fa882a9e3f50c391e3aa08c5c5e72 2023-04-15 02:55:41,264 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing a29fa882a9e3f50c391e3aa08c5c5e72 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-15 02:55:41,264 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a29fa882a9e3f50c391e3aa08c5c5e72, store=A 2023-04-15 02:55:41,264 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-15 02:55:41,264 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a29fa882a9e3f50c391e3aa08c5c5e72, store=B 2023-04-15 02:55:41,264 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-15 02:55:41,264 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a29fa882a9e3f50c391e3aa08c5c5e72, store=C 2023-04-15 02:55:41,264 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-15 02:55:41,295 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9564, totalHeapSize=21976, min timestamp=1681527341282, max timestamp=1681527341293Num uniques 859; 2023-04-15 02:55:41,295 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(163): Suffix data size=39334, new segment data size=9564, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-15 02:55:41,295 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9564, totalHeapSize=21976, min timestamp=1681527341282, max timestamp=1681527341293Num uniques 859; 2023-04-15 02:55:41,295 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-7] regionserver.CompactionPipeline(163): Suffix data size=39334, new segment data size=9564, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-15 02:55:41,295 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9564, totalHeapSize=21976, min timestamp=1681527341282, max timestamp=1681527341293Num uniques 859; 2023-04-15 02:55:41,295 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-9] regionserver.CompactionPipeline(163): Suffix data size=39334, new segment data size=9564, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-15 02:55:41,327 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a29fa882a9e3f50c391e3aa08c5c5e72, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:55:41,327 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34681] ipc.CallRunner(144): callId: 121 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:40192 deadline: 1681527401321, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a29fa882a9e3f50c391e3aa08c5c5e72, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:41,327 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a29fa882a9e3f50c391e3aa08c5c5e72, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:55:41,327 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34681] ipc.CallRunner(144): callId: 113 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:40234 deadline: 1681527401326, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a29fa882a9e3f50c391e3aa08c5c5e72, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:41,331 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a29fa882a9e3f50c391e3aa08c5c5e72, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:55:41,331 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34681] ipc.CallRunner(144): callId: 119 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:40210 deadline: 1681527401327, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a29fa882a9e3f50c391e3aa08c5c5e72, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:41,348 DEBUG [Thread-2739] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-15 02:55:41,348 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-15 02:55:41,349 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-15 02:55:41,350 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-15 02:55:41,350 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-15 02:55:41,350 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-15 02:55:41,350 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-15 02:55:41,350 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-15 02:55:41,350 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-15 02:55:41,351 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:41,351 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:55:41,351 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-15 02:55:41,351 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:41,351 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-15 02:55:41,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-15 02:55:41,352 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:41,352 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-15 02:55:41,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:55:41,352 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:41,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-15 02:55:41,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:55:41,352 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-15 02:55:41,352 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-15 02:55:41,352 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-15 02:55:41,352 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-15 02:55:41,352 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-15 02:55:41,353 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-15 02:55:41,353 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-15 02:55:41,353 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-15 02:55:41,353 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681527331158.a29fa882a9e3f50c391e3aa08c5c5e72. 2023-04-15 02:55:41,353 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681527331158.a29fa882a9e3f50c391e3aa08c5c5e72. started... 2023-04-15 02:55:41,353 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681527331158.a29fa882a9e3f50c391e3aa08c5c5e72. as already flushing 2023-04-15 02:55:41,353 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681527331158.a29fa882a9e3f50c391e3aa08c5c5e72. 2023-04-15 02:55:41,353 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-15 02:55:41,353 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-15 02:55:41,353 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:41,353 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-15 02:55:41,353 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34681,1681527209133' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-15 02:55:41,354 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:41,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:41,355 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:41,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:55:41,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:55:41,355 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:41,355 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-15 02:55:41,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:55:41,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:55:41,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:55:41,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:41,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:55:41,357 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34681,1681527209133' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-15 02:55:41,357 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-15 02:55:41,357 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5a277927[Count = 0] remaining members to acquire global barrier 2023-04-15 02:55:41,357 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:41,358 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:41,358 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:41,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:41,358 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-15 02:55:41,358 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-15 02:55:41,358 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:41,358 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-15 02:55:41,358 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34681,1681527209133' in zk 2023-04-15 02:55:41,359 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:41,359 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-15 02:55:41,359 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-15 02:55:41,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:41,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:55:41,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:55:41,359 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-15 02:55:41,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:55:41,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:55:41,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:55:41,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:41,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:55:41,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:55:41,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:41,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34681,1681527209133': 2023-04-15 02:55:41,361 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34681,1681527209133' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-15 02:55:41,361 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-15 02:55:41,361 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-15 02:55:41,361 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-15 02:55:41,361 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-15 02:55:41,361 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-15 02:55:41,362 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:41,362 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-15 02:55:41,362 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:41,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:41,362 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:41,362 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:41,362 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:41,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:41,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:55:41,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:55:41,363 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-15 02:55:41,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-15 02:55:41,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:55:41,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:41,364 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:41,364 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:55:41,364 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:55:41,364 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:41,364 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:41,364 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:41,364 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:41,364 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-15 02:55:41,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:55:41,364 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-15 02:55:41,364 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-15 02:55:41,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-15 02:55:41,364 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-15 02:55:41,364 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-15 02:55:41,364 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-15 02:55:41,364 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-15 02:55:41,365 DEBUG [Thread-2739] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-15 02:55:41,365 DEBUG [Thread-2739] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-15 02:55:41,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:55:41,368 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-15 02:55:41,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-15 02:55:41,368 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-15 02:55:41,432 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a29fa882a9e3f50c391e3aa08c5c5e72, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:55:41,432 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34681] ipc.CallRunner(144): callId: 115 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:40234 deadline: 1681527401428, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a29fa882a9e3f50c391e3aa08c5c5e72, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:41,432 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a29fa882a9e3f50c391e3aa08c5c5e72, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:55:41,432 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34681] ipc.CallRunner(144): callId: 123 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:40192 deadline: 1681527401428, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a29fa882a9e3f50c391e3aa08c5c5e72, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:41,435 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a29fa882a9e3f50c391e3aa08c5c5e72, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:55:41,435 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34681] ipc.CallRunner(144): callId: 121 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:40210 deadline: 1681527401432, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a29fa882a9e3f50c391e3aa08c5c5e72, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:41,465 DEBUG [Thread-2739] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-15 02:55:41,465 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-15 02:55:41,466 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-15 02:55:41,467 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-15 02:55:41,467 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-15 02:55:41,467 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-15 02:55:41,467 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-15 02:55:41,467 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-15 02:55:41,467 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-15 02:55:41,468 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:41,468 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:55:41,468 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-15 02:55:41,468 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:41,468 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-15 02:55:41,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-15 02:55:41,469 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:41,469 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-15 02:55:41,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:55:41,469 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:41,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-15 02:55:41,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:55:41,469 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-15 02:55:41,469 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-15 02:55:41,469 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-15 02:55:41,470 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-15 02:55:41,470 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-15 02:55:41,470 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-15 02:55:41,470 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-15 02:55:41,470 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-15 02:55:41,470 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681527331158.a29fa882a9e3f50c391e3aa08c5c5e72. 2023-04-15 02:55:41,470 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681527331158.a29fa882a9e3f50c391e3aa08c5c5e72. started... 2023-04-15 02:55:41,470 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681527331158.a29fa882a9e3f50c391e3aa08c5c5e72. as already flushing 2023-04-15 02:55:41,470 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681527331158.a29fa882a9e3f50c391e3aa08c5c5e72. 2023-04-15 02:55:41,470 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-15 02:55:41,470 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-15 02:55:41,470 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:41,470 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-15 02:55:41,470 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34681,1681527209133' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-15 02:55:41,471 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:41,471 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:41,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:41,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:55:41,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:55:41,471 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:41,471 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-15 02:55:41,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:55:41,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:55:41,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:55:41,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:41,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:55:41,472 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34681,1681527209133' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-15 02:55:41,472 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-15 02:55:41,472 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@a25feb[Count = 0] remaining members to acquire global barrier 2023-04-15 02:55:41,472 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:41,473 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:41,473 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:41,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:41,473 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-15 02:55:41,473 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-15 02:55:41,473 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:41,473 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-15 02:55:41,473 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34681,1681527209133' in zk 2023-04-15 02:55:41,474 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:41,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:41,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:55:41,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:55:41,474 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-15 02:55:41,474 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-15 02:55:41,474 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-15 02:55:41,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:55:41,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:55:41,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:55:41,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:41,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:55:41,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:55:41,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:41,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34681,1681527209133': 2023-04-15 02:55:41,476 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34681,1681527209133' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-15 02:55:41,476 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-15 02:55:41,476 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-15 02:55:41,476 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-15 02:55:41,476 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-15 02:55:41,476 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-15 02:55:41,477 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:41,477 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:41,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:41,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:55:41,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:55:41,477 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-15 02:55:41,477 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:41,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:41,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:55:41,477 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:41,477 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-15 02:55:41,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-15 02:55:41,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:41,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:55:41,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:55:41,478 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:41,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:55:41,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:41,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:55:41,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:55:41,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:41,479 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-15 02:55:41,479 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-15 02:55:41,479 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-15 02:55:41,479 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-15 02:55:41,479 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:41,479 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-15 02:55:41,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-15 02:55:41,479 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-15 02:55:41,479 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:55:41,479 DEBUG [Thread-2739] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-15 02:55:41,479 DEBUG [Thread-2739] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-15 02:55:41,479 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:55:41,480 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:41,480 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:41,480 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:41,480 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:41,480 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-15 02:55:41,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-15 02:55:41,480 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-15 02:55:41,580 DEBUG [Thread-2739] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-15 02:55:41,580 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-15 02:55:41,581 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-15 02:55:41,582 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-15 02:55:41,582 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-15 02:55:41,582 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-15 02:55:41,582 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-15 02:55:41,582 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-15 02:55:41,582 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-15 02:55:41,583 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:41,583 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:55:41,583 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-15 02:55:41,583 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:41,583 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-15 02:55:41,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-15 02:55:41,584 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:41,584 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-15 02:55:41,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:55:41,584 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:41,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-15 02:55:41,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:55:41,585 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-15 02:55:41,586 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-15 02:55:41,586 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-15 02:55:41,586 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-15 02:55:41,586 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-15 02:55:41,586 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-15 02:55:41,586 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-15 02:55:41,586 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-15 02:55:41,586 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681527331158.a29fa882a9e3f50c391e3aa08c5c5e72. 2023-04-15 02:55:41,586 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681527331158.a29fa882a9e3f50c391e3aa08c5c5e72. started... 2023-04-15 02:55:41,586 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681527331158.a29fa882a9e3f50c391e3aa08c5c5e72. as already flushing 2023-04-15 02:55:41,586 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681527331158.a29fa882a9e3f50c391e3aa08c5c5e72. 2023-04-15 02:55:41,586 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-15 02:55:41,586 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-15 02:55:41,586 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:41,586 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-15 02:55:41,586 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34681,1681527209133' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-15 02:55:41,587 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:41,587 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:41,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:41,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:55:41,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:55:41,587 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:41,587 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-15 02:55:41,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:55:41,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:55:41,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:55:41,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:41,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:55:41,588 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34681,1681527209133' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-15 02:55:41,588 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-15 02:55:41,588 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4d5bf45[Count = 0] remaining members to acquire global barrier 2023-04-15 02:55:41,588 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:41,590 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:41,590 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:41,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:41,590 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:41,590 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-15 02:55:41,590 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-15 02:55:41,590 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-15 02:55:41,590 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34681,1681527209133' in zk 2023-04-15 02:55:41,591 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:41,591 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-15 02:55:41,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:41,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:55:41,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:55:41,591 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-15 02:55:41,591 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-15 02:55:41,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:55:41,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:55:41,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:55:41,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:41,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:55:41,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:55:41,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:41,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34681,1681527209133': 2023-04-15 02:55:41,593 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34681,1681527209133' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-15 02:55:41,593 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-15 02:55:41,593 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-15 02:55:41,593 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-15 02:55:41,593 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-15 02:55:41,593 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-15 02:55:41,594 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:41,594 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:41,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:41,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:55:41,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:55:41,594 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:41,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:41,594 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-15 02:55:41,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:55:41,594 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-15 02:55:41,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-15 02:55:41,594 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:41,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:41,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:55:41,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:55:41,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:55:41,595 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:41,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:41,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:55:41,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:55:41,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:41,598 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:41,598 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-15 02:55:41,598 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:55:41,598 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-15 02:55:41,598 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-15 02:55:41,598 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-15 02:55:41,598 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-15 02:55:41,598 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:55:41,598 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-15 02:55:41,598 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-15 02:55:41,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-15 02:55:41,598 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:41,598 DEBUG [Thread-2739] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-15 02:55:41,598 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:41,598 DEBUG [Thread-2739] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-15 02:55:41,598 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:41,598 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-15 02:55:41,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-15 02:55:41,598 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:41,637 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a29fa882a9e3f50c391e3aa08c5c5e72, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:55:41,637 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34681] ipc.CallRunner(144): callId: 125 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:40192 deadline: 1681527401633, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a29fa882a9e3f50c391e3aa08c5c5e72, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:41,637 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a29fa882a9e3f50c391e3aa08c5c5e72, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:55:41,637 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34681] ipc.CallRunner(144): callId: 117 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:40234 deadline: 1681527401634, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a29fa882a9e3f50c391e3aa08c5c5e72, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:41,637 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a29fa882a9e3f50c391e3aa08c5c5e72, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:55:41,637 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34681] ipc.CallRunner(144): callId: 123 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:40210 deadline: 1681527401636, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a29fa882a9e3f50c391e3aa08c5c5e72, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:41,684 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=222 (bloomFilter=true), to=hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/a29fa882a9e3f50c391e3aa08c5c5e72/.tmp/A/1825e46b7b414112bf098f0a6f0c0c09 2023-04-15 02:55:41,698 DEBUG [Thread-2739] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-15 02:55:41,699 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-15 02:55:41,700 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-15 02:55:41,701 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-15 02:55:41,701 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-15 02:55:41,701 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-15 02:55:41,701 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-15 02:55:41,701 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-15 02:55:41,701 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-15 02:55:41,702 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:41,702 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:55:41,702 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-15 02:55:41,702 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:41,702 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-15 02:55:41,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-15 02:55:41,703 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:41,703 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-15 02:55:41,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:55:41,703 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:41,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-15 02:55:41,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:55:41,703 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-15 02:55:41,703 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-15 02:55:41,703 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-15 02:55:41,703 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-15 02:55:41,703 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-15 02:55:41,703 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-15 02:55:41,704 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-15 02:55:41,704 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-15 02:55:41,704 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681527331158.a29fa882a9e3f50c391e3aa08c5c5e72. 2023-04-15 02:55:41,704 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681527331158.a29fa882a9e3f50c391e3aa08c5c5e72. started... 2023-04-15 02:55:41,704 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681527331158.a29fa882a9e3f50c391e3aa08c5c5e72. as already flushing 2023-04-15 02:55:41,704 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681527331158.a29fa882a9e3f50c391e3aa08c5c5e72. 2023-04-15 02:55:41,704 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-15 02:55:41,704 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-15 02:55:41,704 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:41,704 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-15 02:55:41,704 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34681,1681527209133' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-15 02:55:41,705 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:41,705 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:41,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:41,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:55:41,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:55:41,705 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:41,705 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-15 02:55:41,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:55:41,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:55:41,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:55:41,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:41,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:55:41,706 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34681,1681527209133' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-15 02:55:41,706 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-15 02:55:41,706 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5eeffe67[Count = 0] remaining members to acquire global barrier 2023-04-15 02:55:41,706 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:41,706 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:41,706 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:41,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:41,706 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-15 02:55:41,706 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-15 02:55:41,706 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34681,1681527209133' in zk 2023-04-15 02:55:41,707 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:41,707 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-15 02:55:41,708 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-15 02:55:41,708 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:41,708 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-15 02:55:41,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:41,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:55:41,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:55:41,708 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-15 02:55:41,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:55:41,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:55:41,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:55:41,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:41,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:55:41,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:55:41,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:41,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34681,1681527209133': 2023-04-15 02:55:41,709 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34681,1681527209133' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-15 02:55:41,709 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-15 02:55:41,709 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-15 02:55:41,709 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-15 02:55:41,709 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-15 02:55:41,710 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-15 02:55:41,710 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:41,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:41,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:55:41,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:55:41,710 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:41,710 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-15 02:55:41,710 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:41,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:41,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:55:41,711 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:41,711 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-15 02:55:41,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-15 02:55:41,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:55:41,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:41,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:55:41,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:55:41,712 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:41,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:41,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:55:41,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:55:41,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:41,713 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:41,713 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-15 02:55:41,713 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:55:41,713 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-15 02:55:41,713 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-15 02:55:41,713 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-15 02:55:41,713 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-15 02:55:41,713 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-15 02:55:41,713 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-15 02:55:41,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-15 02:55:41,713 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:55:41,713 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:41,713 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:41,713 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:41,713 DEBUG [Thread-2739] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-15 02:55:41,713 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:41,713 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-15 02:55:41,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-15 02:55:41,713 DEBUG [Thread-2739] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-15 02:55:41,813 DEBUG [Thread-2739] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-15 02:55:41,814 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-15 02:55:41,815 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-15 02:55:41,816 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-15 02:55:41,816 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-15 02:55:41,816 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-15 02:55:41,816 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-15 02:55:41,816 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-15 02:55:41,816 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-15 02:55:41,816 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:41,816 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:55:41,817 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-15 02:55:41,817 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:41,817 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-15 02:55:41,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-15 02:55:41,817 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:41,817 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-15 02:55:41,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:55:41,818 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:41,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-15 02:55:41,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:55:41,818 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-15 02:55:41,818 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-15 02:55:41,818 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-15 02:55:41,818 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-15 02:55:41,818 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-15 02:55:41,818 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-15 02:55:41,818 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-15 02:55:41,819 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-15 02:55:41,819 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681527331158.a29fa882a9e3f50c391e3aa08c5c5e72. 2023-04-15 02:55:41,819 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681527331158.a29fa882a9e3f50c391e3aa08c5c5e72. started... 2023-04-15 02:55:41,819 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681527331158.a29fa882a9e3f50c391e3aa08c5c5e72. as already flushing 2023-04-15 02:55:41,819 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681527331158.a29fa882a9e3f50c391e3aa08c5c5e72. 2023-04-15 02:55:41,819 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-15 02:55:41,819 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-15 02:55:41,819 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:41,819 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-15 02:55:41,819 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34681,1681527209133' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-15 02:55:41,820 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:41,820 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:41,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:41,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:55:41,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:55:41,820 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:41,820 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-15 02:55:41,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:55:41,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:55:41,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:55:41,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:41,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:55:41,821 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34681,1681527209133' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-15 02:55:41,821 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-15 02:55:41,821 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3fe05adc[Count = 0] remaining members to acquire global barrier 2023-04-15 02:55:41,821 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:41,821 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:41,821 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:41,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:41,822 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-15 02:55:41,822 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-15 02:55:41,822 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34681,1681527209133' in zk 2023-04-15 02:55:41,822 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:41,822 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-15 02:55:41,822 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:41,822 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-15 02:55:41,822 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-15 02:55:41,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:41,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:55:41,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:55:41,822 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-15 02:55:41,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:55:41,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:55:41,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:55:41,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:41,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:55:41,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:55:41,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:41,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34681,1681527209133': 2023-04-15 02:55:41,824 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34681,1681527209133' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-15 02:55:41,824 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-15 02:55:41,824 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-15 02:55:41,824 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-15 02:55:41,824 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-15 02:55:41,824 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-15 02:55:41,825 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:41,825 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:41,825 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-15 02:55:41,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:41,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:55:41,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:55:41,825 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:41,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:41,825 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:41,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:55:41,825 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-15 02:55:41,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-15 02:55:41,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:55:41,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:41,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:55:41,827 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:41,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:55:41,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:41,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:55:41,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:55:41,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:41,828 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-15 02:55:41,828 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-15 02:55:41,828 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-15 02:55:41,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-15 02:55:41,828 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-15 02:55:41,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-15 02:55:41,829 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:41,829 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:55:41,829 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:55:41,829 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:41,829 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:41,829 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:41,829 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-15 02:55:41,829 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-15 02:55:41,829 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-15 02:55:41,829 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:41,829 DEBUG [Thread-2739] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-15 02:55:41,829 DEBUG [Thread-2739] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-15 02:55:41,830 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-15 02:55:41,929 DEBUG [Thread-2739] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-15 02:55:41,930 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-15 02:55:41,931 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-15 02:55:41,932 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-15 02:55:41,932 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-15 02:55:41,932 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-15 02:55:41,933 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-15 02:55:41,933 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-15 02:55:41,933 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-15 02:55:41,933 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:41,933 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:55:41,934 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-15 02:55:41,934 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:41,934 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-15 02:55:41,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-15 02:55:41,934 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:41,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:55:41,934 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-15 02:55:41,934 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:41,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-15 02:55:41,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:55:41,934 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-15 02:55:41,935 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-15 02:55:41,935 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-15 02:55:41,935 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-15 02:55:41,935 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-15 02:55:41,935 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-15 02:55:41,935 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-15 02:55:41,935 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-15 02:55:41,935 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681527331158.a29fa882a9e3f50c391e3aa08c5c5e72. 2023-04-15 02:55:41,935 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681527331158.a29fa882a9e3f50c391e3aa08c5c5e72. started... 2023-04-15 02:55:41,935 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681527331158.a29fa882a9e3f50c391e3aa08c5c5e72. as already flushing 2023-04-15 02:55:41,935 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681527331158.a29fa882a9e3f50c391e3aa08c5c5e72. 2023-04-15 02:55:41,935 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-15 02:55:41,935 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-15 02:55:41,935 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:41,935 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-15 02:55:41,935 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34681,1681527209133' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-15 02:55:41,936 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:41,936 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:41,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:41,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:55:41,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:55:41,936 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:41,936 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-15 02:55:41,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:55:41,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:55:41,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:55:41,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:41,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:55:41,937 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34681,1681527209133' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-15 02:55:41,937 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2daf17cc[Count = 0] remaining members to acquire global barrier 2023-04-15 02:55:41,937 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-15 02:55:41,937 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:41,938 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:41,938 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:41,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:41,938 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-15 02:55:41,938 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-15 02:55:41,938 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34681,1681527209133' in zk 2023-04-15 02:55:41,938 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:41,938 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-15 02:55:41,939 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-15 02:55:41,939 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-15 02:55:41,939 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-15 02:55:41,939 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:41,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:41,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:55:41,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:55:41,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:55:41,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:55:41,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:55:41,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:41,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:55:41,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:55:41,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:41,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34681,1681527209133': 2023-04-15 02:55:41,941 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34681,1681527209133' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-15 02:55:41,941 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-15 02:55:41,941 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-15 02:55:41,941 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-15 02:55:41,941 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-15 02:55:41,941 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-15 02:55:41,941 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a29fa882a9e3f50c391e3aa08c5c5e72, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:55:41,942 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34681] ipc.CallRunner(144): callId: 125 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:40210 deadline: 1681527401938, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a29fa882a9e3f50c391e3aa08c5c5e72, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:41,942 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a29fa882a9e3f50c391e3aa08c5c5e72, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:55:41,942 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34681] ipc.CallRunner(144): callId: 127 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:40192 deadline: 1681527401939, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a29fa882a9e3f50c391e3aa08c5c5e72, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:41,942 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:41,942 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:41,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:41,942 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:41,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:41,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:55:41,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:55:41,942 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-15 02:55:41,943 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-15 02:55:41,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-15 02:55:41,943 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:41,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:55:41,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:41,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:55:41,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:55:41,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:55:41,944 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:41,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:41,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:55:41,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:55:41,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:41,945 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:41,945 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:55:41,945 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-15 02:55:41,945 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:55:41,945 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-15 02:55:41,945 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:41,945 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-15 02:55:41,945 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-15 02:55:41,945 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-15 02:55:41,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-15 02:55:41,945 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-15 02:55:41,945 DEBUG [Thread-2739] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-15 02:55:41,945 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a29fa882a9e3f50c391e3aa08c5c5e72, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:55:41,945 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-15 02:55:41,945 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34681] ipc.CallRunner(144): callId: 119 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:40234 deadline: 1681527401939, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a29fa882a9e3f50c391e3aa08c5c5e72, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:41,945 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:41,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-15 02:55:41,945 DEBUG [Thread-2739] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-15 02:55:41,945 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-15 02:55:41,945 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:41,946 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:42,046 DEBUG [Thread-2739] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-15 02:55:42,046 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-15 02:55:42,047 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-15 02:55:42,048 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-15 02:55:42,048 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-15 02:55:42,048 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-15 02:55:42,048 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-15 02:55:42,048 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-15 02:55:42,048 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-15 02:55:42,048 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:42,048 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:55:42,049 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-15 02:55:42,049 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:42,049 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-15 02:55:42,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-15 02:55:42,049 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:42,049 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-15 02:55:42,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:55:42,050 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:42,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-15 02:55:42,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:55:42,050 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-15 02:55:42,050 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-15 02:55:42,050 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-15 02:55:42,050 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-15 02:55:42,050 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-15 02:55:42,050 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-15 02:55:42,050 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-15 02:55:42,051 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-15 02:55:42,051 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681527331158.a29fa882a9e3f50c391e3aa08c5c5e72. 2023-04-15 02:55:42,051 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681527331158.a29fa882a9e3f50c391e3aa08c5c5e72. started... 2023-04-15 02:55:42,051 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681527331158.a29fa882a9e3f50c391e3aa08c5c5e72. as already flushing 2023-04-15 02:55:42,051 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681527331158.a29fa882a9e3f50c391e3aa08c5c5e72. 2023-04-15 02:55:42,051 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-15 02:55:42,051 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-15 02:55:42,051 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:42,051 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-15 02:55:42,051 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34681,1681527209133' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-15 02:55:42,052 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:42,052 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:42,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:42,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:55:42,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:55:42,052 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:42,052 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-15 02:55:42,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:55:42,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:55:42,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:55:42,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:42,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:55:42,053 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34681,1681527209133' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-15 02:55:42,053 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-15 02:55:42,053 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4a9baab[Count = 0] remaining members to acquire global barrier 2023-04-15 02:55:42,053 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:42,054 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:42,054 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:42,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:42,054 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-15 02:55:42,054 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:42,054 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-15 02:55:42,054 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-15 02:55:42,054 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34681,1681527209133' in zk 2023-04-15 02:55:42,055 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-15 02:55:42,055 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-15 02:55:42,055 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:42,055 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-15 02:55:42,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:42,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:55:42,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:55:42,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:55:42,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:55:42,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:55:42,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:42,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:55:42,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:55:42,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:42,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34681,1681527209133': 2023-04-15 02:55:42,057 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34681,1681527209133' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-15 02:55:42,057 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-15 02:55:42,057 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-15 02:55:42,057 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-15 02:55:42,057 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-15 02:55:42,057 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-15 02:55:42,058 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:42,058 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-15 02:55:42,058 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:42,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:42,058 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:42,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:42,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:55:42,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:55:42,058 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-15 02:55:42,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-15 02:55:42,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:42,058 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:42,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:55:42,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:55:42,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:55:42,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:55:42,059 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:42,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:42,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:55:42,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:55:42,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:42,061 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-15 02:55:42,061 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-15 02:55:42,061 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-15 02:55:42,061 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-15 02:55:42,061 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:42,061 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-15 02:55:42,061 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-15 02:55:42,061 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-15 02:55:42,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-15 02:55:42,061 DEBUG [Thread-2739] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-15 02:55:42,061 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:55:42,061 DEBUG [Thread-2739] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-15 02:55:42,061 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:55:42,061 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:42,061 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:42,061 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-15 02:55:42,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-15 02:55:42,061 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:42,061 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:42,092 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=222 (bloomFilter=true), to=hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/a29fa882a9e3f50c391e3aa08c5c5e72/.tmp/B/7feccce09cba4773982cddef4fc1b5f3 2023-04-15 02:55:42,161 DEBUG [Thread-2739] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-15 02:55:42,162 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-15 02:55:42,163 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-15 02:55:42,164 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-15 02:55:42,164 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-15 02:55:42,164 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-15 02:55:42,164 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-15 02:55:42,164 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-15 02:55:42,164 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-15 02:55:42,164 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:42,164 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:55:42,165 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-15 02:55:42,165 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:42,165 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-15 02:55:42,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-15 02:55:42,165 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:42,165 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-15 02:55:42,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:55:42,166 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:42,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-15 02:55:42,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:55:42,166 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-15 02:55:42,166 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-15 02:55:42,166 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-15 02:55:42,166 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-15 02:55:42,166 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-15 02:55:42,166 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-15 02:55:42,166 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-15 02:55:42,166 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-15 02:55:42,166 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681527331158.a29fa882a9e3f50c391e3aa08c5c5e72. 2023-04-15 02:55:42,166 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681527331158.a29fa882a9e3f50c391e3aa08c5c5e72. started... 2023-04-15 02:55:42,167 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681527331158.a29fa882a9e3f50c391e3aa08c5c5e72. as already flushing 2023-04-15 02:55:42,167 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681527331158.a29fa882a9e3f50c391e3aa08c5c5e72. 2023-04-15 02:55:42,167 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-15 02:55:42,167 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-15 02:55:42,167 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:42,167 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-15 02:55:42,167 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34681,1681527209133' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-15 02:55:42,167 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:42,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:42,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:55:42,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:55:42,167 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:42,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:55:42,168 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:42,168 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-15 02:55:42,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:55:42,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:55:42,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:42,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:55:42,169 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34681,1681527209133' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-15 02:55:42,169 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2594709f[Count = 0] remaining members to acquire global barrier 2023-04-15 02:55:42,169 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-15 02:55:42,169 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:42,169 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:42,169 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:42,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:42,169 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-15 02:55:42,169 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-15 02:55:42,169 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:42,169 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-15 02:55:42,169 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34681,1681527209133' in zk 2023-04-15 02:55:42,170 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:42,170 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-15 02:55:42,170 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-15 02:55:42,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:42,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:55:42,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:55:42,170 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-15 02:55:42,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:55:42,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:55:42,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:55:42,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:42,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:55:42,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:55:42,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:42,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34681,1681527209133': 2023-04-15 02:55:42,172 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34681,1681527209133' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-15 02:55:42,172 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-15 02:55:42,172 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-15 02:55:42,172 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-15 02:55:42,172 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-15 02:55:42,172 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-15 02:55:42,173 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:42,173 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:42,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:42,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:55:42,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:55:42,173 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-15 02:55:42,173 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:42,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:42,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:55:42,173 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:42,174 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-15 02:55:42,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-15 02:55:42,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:55:42,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:42,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:55:42,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:55:42,175 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:42,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:42,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:55:42,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:55:42,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:42,176 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-15 02:55:42,176 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-15 02:55:42,176 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:42,176 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:55:42,176 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:55:42,176 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:42,176 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:42,176 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:42,176 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:42,176 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-15 02:55:42,176 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-15 02:55:42,176 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-15 02:55:42,176 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-15 02:55:42,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-15 02:55:42,176 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-15 02:55:42,177 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-15 02:55:42,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-15 02:55:42,177 DEBUG [Thread-2739] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-15 02:55:42,177 DEBUG [Thread-2739] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-15 02:55:42,277 DEBUG [Thread-2739] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-15 02:55:42,277 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-15 02:55:42,279 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-15 02:55:42,279 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-15 02:55:42,279 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-15 02:55:42,279 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-15 02:55:42,280 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-15 02:55:42,280 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-15 02:55:42,280 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-15 02:55:42,280 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:42,280 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:55:42,281 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:42,281 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-15 02:55:42,281 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-15 02:55:42,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-15 02:55:42,281 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:42,281 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-15 02:55:42,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:55:42,281 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:42,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-15 02:55:42,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:55:42,281 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-15 02:55:42,282 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-15 02:55:42,282 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-15 02:55:42,282 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-15 02:55:42,282 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-15 02:55:42,282 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-15 02:55:42,282 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-15 02:55:42,282 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-15 02:55:42,282 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681527331158.a29fa882a9e3f50c391e3aa08c5c5e72. 2023-04-15 02:55:42,282 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681527331158.a29fa882a9e3f50c391e3aa08c5c5e72. started... 2023-04-15 02:55:42,282 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681527331158.a29fa882a9e3f50c391e3aa08c5c5e72. as already flushing 2023-04-15 02:55:42,282 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681527331158.a29fa882a9e3f50c391e3aa08c5c5e72. 2023-04-15 02:55:42,282 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-15 02:55:42,282 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-15 02:55:42,282 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:42,282 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-15 02:55:42,282 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34681,1681527209133' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-15 02:55:42,283 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:42,283 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:42,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:42,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:55:42,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:55:42,283 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:42,283 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-15 02:55:42,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:55:42,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:55:42,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:55:42,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:42,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:55:42,284 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34681,1681527209133' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-15 02:55:42,284 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-15 02:55:42,284 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@181fe84b[Count = 0] remaining members to acquire global barrier 2023-04-15 02:55:42,284 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:42,285 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:42,285 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:42,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:42,285 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-15 02:55:42,285 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-15 02:55:42,285 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:42,285 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-15 02:55:42,285 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34681,1681527209133' in zk 2023-04-15 02:55:42,286 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-15 02:55:42,286 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:42,286 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-15 02:55:42,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:42,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:55:42,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:55:42,286 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-15 02:55:42,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:55:42,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:55:42,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:55:42,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:42,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:55:42,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:55:42,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:42,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34681,1681527209133': 2023-04-15 02:55:42,287 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34681,1681527209133' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-15 02:55:42,287 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-15 02:55:42,287 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-15 02:55:42,287 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-15 02:55:42,287 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-15 02:55:42,287 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-15 02:55:42,288 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:42,288 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:42,288 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-15 02:55:42,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:42,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:55:42,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:55:42,289 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:42,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:42,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:55:42,289 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-15 02:55:42,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-15 02:55:42,289 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:42,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:55:42,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:42,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:55:42,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:55:42,290 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:42,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:42,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:55:42,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:55:42,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:42,292 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-15 02:55:42,292 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:42,292 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-15 02:55:42,292 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:55:42,292 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-15 02:55:42,292 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-15 02:55:42,292 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:55:42,292 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:42,292 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:42,292 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:42,292 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:42,292 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-15 02:55:42,293 DEBUG [Thread-2739] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-15 02:55:42,292 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-15 02:55:42,293 DEBUG [Thread-2739] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-15 02:55:42,292 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-15 02:55:42,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-15 02:55:42,293 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-15 02:55:42,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-15 02:55:42,393 DEBUG [Thread-2739] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-15 02:55:42,393 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-15 02:55:42,395 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-15 02:55:42,396 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-15 02:55:42,397 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-15 02:55:42,397 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-15 02:55:42,397 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-15 02:55:42,397 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-15 02:55:42,397 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-15 02:55:42,397 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:42,397 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:55:42,398 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-15 02:55:42,398 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:42,398 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-15 02:55:42,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-15 02:55:42,398 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:42,398 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-15 02:55:42,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:55:42,398 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:42,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-15 02:55:42,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:55:42,399 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-15 02:55:42,399 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-15 02:55:42,399 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-15 02:55:42,399 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-15 02:55:42,399 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-15 02:55:42,399 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-15 02:55:42,399 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-15 02:55:42,399 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-15 02:55:42,399 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681527331158.a29fa882a9e3f50c391e3aa08c5c5e72. 2023-04-15 02:55:42,399 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681527331158.a29fa882a9e3f50c391e3aa08c5c5e72. started... 2023-04-15 02:55:42,399 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681527331158.a29fa882a9e3f50c391e3aa08c5c5e72. as already flushing 2023-04-15 02:55:42,399 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681527331158.a29fa882a9e3f50c391e3aa08c5c5e72. 2023-04-15 02:55:42,400 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-15 02:55:42,400 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-15 02:55:42,400 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:42,400 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-15 02:55:42,400 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34681,1681527209133' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-15 02:55:42,400 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:42,400 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:42,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:42,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:55:42,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:55:42,400 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:42,401 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-15 02:55:42,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:55:42,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:55:42,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:55:42,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:42,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:55:42,401 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34681,1681527209133' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-15 02:55:42,401 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4ddb8df3[Count = 0] remaining members to acquire global barrier 2023-04-15 02:55:42,401 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-15 02:55:42,401 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:42,402 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:42,402 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:42,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:42,402 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-15 02:55:42,402 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-15 02:55:42,402 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34681,1681527209133' in zk 2023-04-15 02:55:42,402 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:42,402 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-15 02:55:42,403 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:42,403 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-15 02:55:42,403 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-15 02:55:42,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:42,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:55:42,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:55:42,403 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-15 02:55:42,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:55:42,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:55:42,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:55:42,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:42,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:55:42,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:55:42,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:42,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34681,1681527209133': 2023-04-15 02:55:42,405 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34681,1681527209133' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-15 02:55:42,405 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-15 02:55:42,405 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-15 02:55:42,405 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-15 02:55:42,405 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-15 02:55:42,405 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-15 02:55:42,406 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:42,406 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:42,406 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:42,406 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-15 02:55:42,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:42,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:42,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:55:42,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:55:42,406 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-15 02:55:42,406 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:42,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-15 02:55:42,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:55:42,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:42,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:55:42,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:55:42,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:55:42,407 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:42,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:42,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:55:42,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:55:42,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:42,408 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:42,408 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:55:42,408 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-15 02:55:42,408 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:55:42,415 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-15 02:55:42,415 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:42,408 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-15 02:55:42,416 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-15 02:55:42,416 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:42,416 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:42,416 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:42,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-15 02:55:42,416 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-15 02:55:42,416 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-15 02:55:42,416 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-15 02:55:42,416 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-15 02:55:42,416 DEBUG [Thread-2739] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-15 02:55:42,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-15 02:55:42,416 DEBUG [Thread-2739] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-15 02:55:42,453 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a29fa882a9e3f50c391e3aa08c5c5e72, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:55:42,453 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34681] ipc.CallRunner(144): callId: 127 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:40210 deadline: 1681527402444, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a29fa882a9e3f50c391e3aa08c5c5e72, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:42,453 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a29fa882a9e3f50c391e3aa08c5c5e72, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:55:42,453 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34681] ipc.CallRunner(144): callId: 129 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:40192 deadline: 1681527402444, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a29fa882a9e3f50c391e3aa08c5c5e72, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:42,455 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a29fa882a9e3f50c391e3aa08c5c5e72, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:55:42,456 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34681] ipc.CallRunner(144): callId: 121 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:40234 deadline: 1681527402450, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a29fa882a9e3f50c391e3aa08c5c5e72, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:42,500 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=222 (bloomFilter=true), to=hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/a29fa882a9e3f50c391e3aa08c5c5e72/.tmp/C/1290cf8f5e884c469991519a85d516b9 2023-04-15 02:55:42,503 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/a29fa882a9e3f50c391e3aa08c5c5e72/.tmp/A/1825e46b7b414112bf098f0a6f0c0c09 as hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/a29fa882a9e3f50c391e3aa08c5c5e72/A/1825e46b7b414112bf098f0a6f0c0c09 2023-04-15 02:55:42,505 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/a29fa882a9e3f50c391e3aa08c5c5e72/A/1825e46b7b414112bf098f0a6f0c0c09, entries=200, sequenceid=222, filesize=14.1 K 2023-04-15 02:55:42,506 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/a29fa882a9e3f50c391e3aa08c5c5e72/.tmp/B/7feccce09cba4773982cddef4fc1b5f3 as hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/a29fa882a9e3f50c391e3aa08c5c5e72/B/7feccce09cba4773982cddef4fc1b5f3 2023-04-15 02:55:42,509 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/a29fa882a9e3f50c391e3aa08c5c5e72/B/7feccce09cba4773982cddef4fc1b5f3, entries=100, sequenceid=222, filesize=9.4 K 2023-04-15 02:55:42,509 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/a29fa882a9e3f50c391e3aa08c5c5e72/.tmp/C/1290cf8f5e884c469991519a85d516b9 as hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/a29fa882a9e3f50c391e3aa08c5c5e72/C/1290cf8f5e884c469991519a85d516b9 2023-04-15 02:55:42,512 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/a29fa882a9e3f50c391e3aa08c5c5e72/C/1290cf8f5e884c469991519a85d516b9, entries=100, sequenceid=222, filesize=9.4 K 2023-04-15 02:55:42,512 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=134.18 KB/137400 for a29fa882a9e3f50c391e3aa08c5c5e72 in 1248ms, sequenceid=222, compaction requested=true 2023-04-15 02:55:42,512 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for a29fa882a9e3f50c391e3aa08c5c5e72: 2023-04-15 02:55:42,513 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-15 02:55:42,513 DEBUG [RS:0;jenkins-hbase20:34681-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-15 02:55:42,513 DEBUG [RS:0;jenkins-hbase20:34681-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-15 02:55:42,513 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-15 02:55:42,513 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-15 02:55:42,513 DEBUG [RS:0;jenkins-hbase20:34681-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 38870 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-15 02:55:42,513 DEBUG [RS:0;jenkins-hbase20:34681-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 34090 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-15 02:55:42,513 DEBUG [RS:0;jenkins-hbase20:34681-shortCompactions-0] regionserver.HStore(1912): a29fa882a9e3f50c391e3aa08c5c5e72/A is initiating minor compaction (all files) 2023-04-15 02:55:42,513 DEBUG [RS:0;jenkins-hbase20:34681-longCompactions-0] regionserver.HStore(1912): a29fa882a9e3f50c391e3aa08c5c5e72/B is initiating minor compaction (all files) 2023-04-15 02:55:42,514 INFO [RS:0;jenkins-hbase20:34681-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of a29fa882a9e3f50c391e3aa08c5c5e72/A in TestAcidGuarantees,,1681527331158.a29fa882a9e3f50c391e3aa08c5c5e72. 2023-04-15 02:55:42,514 INFO [RS:0;jenkins-hbase20:34681-longCompactions-0] regionserver.HRegion(2259): Starting compaction of a29fa882a9e3f50c391e3aa08c5c5e72/B in TestAcidGuarantees,,1681527331158.a29fa882a9e3f50c391e3aa08c5c5e72. 2023-04-15 02:55:42,514 INFO [RS:0;jenkins-hbase20:34681-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/a29fa882a9e3f50c391e3aa08c5c5e72/A/4ab24298f79e488abab5ebc46292e6b3, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/a29fa882a9e3f50c391e3aa08c5c5e72/A/d2705aa6585d42dba39e3bbfd4abf7ef, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/a29fa882a9e3f50c391e3aa08c5c5e72/A/1825e46b7b414112bf098f0a6f0c0c09] into tmpdir=hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/a29fa882a9e3f50c391e3aa08c5c5e72/.tmp, totalSize=38.0 K 2023-04-15 02:55:42,514 INFO [RS:0;jenkins-hbase20:34681-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/a29fa882a9e3f50c391e3aa08c5c5e72/B/af8ce0b6d48546668e62dac87f2fbbb9, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/a29fa882a9e3f50c391e3aa08c5c5e72/B/28da064938ee4d62a7003f2600f11230, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/a29fa882a9e3f50c391e3aa08c5c5e72/B/7feccce09cba4773982cddef4fc1b5f3] into tmpdir=hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/a29fa882a9e3f50c391e3aa08c5c5e72/.tmp, totalSize=33.3 K 2023-04-15 02:55:42,514 DEBUG [RS:0;jenkins-hbase20:34681-shortCompactions-0] compactions.Compactor(207): Compacting 4ab24298f79e488abab5ebc46292e6b3, keycount=150, bloomtype=ROW, size=12.1 K, encoding=NONE, compression=NONE, seqNum=191, earliestPutTs=1681527337970 2023-04-15 02:55:42,514 DEBUG [RS:0;jenkins-hbase20:34681-longCompactions-0] compactions.Compactor(207): Compacting af8ce0b6d48546668e62dac87f2fbbb9, keycount=150, bloomtype=ROW, size=12.1 K, encoding=NONE, compression=NONE, seqNum=191, earliestPutTs=1681527337970 2023-04-15 02:55:42,514 DEBUG [RS:0;jenkins-hbase20:34681-shortCompactions-0] compactions.Compactor(207): Compacting d2705aa6585d42dba39e3bbfd4abf7ef, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=210, earliestPutTs=1681527339129 2023-04-15 02:55:42,514 DEBUG [RS:0;jenkins-hbase20:34681-longCompactions-0] compactions.Compactor(207): Compacting 28da064938ee4d62a7003f2600f11230, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=210, earliestPutTs=1681527339129 2023-04-15 02:55:42,514 DEBUG [RS:0;jenkins-hbase20:34681-shortCompactions-0] compactions.Compactor(207): Compacting 1825e46b7b414112bf098f0a6f0c0c09, keycount=200, bloomtype=ROW, size=14.1 K, encoding=NONE, compression=NONE, seqNum=222, earliestPutTs=1681527341259 2023-04-15 02:55:42,514 DEBUG [RS:0;jenkins-hbase20:34681-longCompactions-0] compactions.Compactor(207): Compacting 7feccce09cba4773982cddef4fc1b5f3, keycount=100, bloomtype=ROW, size=9.4 K, encoding=NONE, compression=NONE, seqNum=222, earliestPutTs=1681527341259 2023-04-15 02:55:42,516 DEBUG [Thread-2739] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-15 02:55:42,517 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-15 02:55:42,518 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-15 02:55:42,520 INFO [RS:0;jenkins-hbase20:34681-longCompactions-0] throttle.PressureAwareThroughputController(145): a29fa882a9e3f50c391e3aa08c5c5e72#B#compaction#387 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-15 02:55:42,520 INFO [RS:0;jenkins-hbase20:34681-shortCompactions-0] throttle.PressureAwareThroughputController(145): a29fa882a9e3f50c391e3aa08c5c5e72#A#compaction#386 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-15 02:55:42,520 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-15 02:55:42,520 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-15 02:55:42,520 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-15 02:55:42,520 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-15 02:55:42,520 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-15 02:55:42,520 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-15 02:55:42,521 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:42,521 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:55:42,521 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:42,521 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-15 02:55:42,521 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-15 02:55:42,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-15 02:55:42,522 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:42,522 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-15 02:55:42,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:55:42,522 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:42,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-15 02:55:42,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:55:42,522 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-15 02:55:42,524 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-15 02:55:42,524 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-15 02:55:42,524 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-15 02:55:42,524 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-15 02:55:42,524 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-15 02:55:42,524 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681527331158.a29fa882a9e3f50c391e3aa08c5c5e72. 2023-04-15 02:55:42,524 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-15 02:55:42,524 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681527331158.a29fa882a9e3f50c391e3aa08c5c5e72. started... 2023-04-15 02:55:42,524 INFO [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing a29fa882a9e3f50c391e3aa08c5c5e72 3/3 column families, dataSize=134.18 KB heapSize=380.30 KB 2023-04-15 02:55:42,525 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a29fa882a9e3f50c391e3aa08c5c5e72, store=A 2023-04-15 02:55:42,525 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-15 02:55:42,525 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a29fa882a9e3f50c391e3aa08c5c5e72, store=B 2023-04-15 02:55:42,525 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-15 02:55:42,525 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a29fa882a9e3f50c391e3aa08c5c5e72, store=C 2023-04-15 02:55:42,525 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-15 02:55:42,530 DEBUG [RS:0;jenkins-hbase20:34681-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/a29fa882a9e3f50c391e3aa08c5c5e72/.tmp/A/bf1a5b878def417280c39297674887c5 as hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/a29fa882a9e3f50c391e3aa08c5c5e72/A/bf1a5b878def417280c39297674887c5 2023-04-15 02:55:42,534 INFO [RS:0;jenkins-hbase20:34681-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in a29fa882a9e3f50c391e3aa08c5c5e72/A of a29fa882a9e3f50c391e3aa08c5c5e72 into bf1a5b878def417280c39297674887c5(size=12.2 K), total size for store is 12.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-15 02:55:42,534 DEBUG [RS:0;jenkins-hbase20:34681-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for a29fa882a9e3f50c391e3aa08c5c5e72: 2023-04-15 02:55:42,534 INFO [RS:0;jenkins-hbase20:34681-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681527331158.a29fa882a9e3f50c391e3aa08c5c5e72., storeName=a29fa882a9e3f50c391e3aa08c5c5e72/A, priority=13, startTime=1681527342513; duration=0sec 2023-04-15 02:55:42,534 DEBUG [RS:0;jenkins-hbase20:34681-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-15 02:55:42,534 DEBUG [RS:0;jenkins-hbase20:34681-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-15 02:55:42,536 DEBUG [RS:0;jenkins-hbase20:34681-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 34090 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-15 02:55:42,536 DEBUG [RS:0;jenkins-hbase20:34681-shortCompactions-0] regionserver.HStore(1912): a29fa882a9e3f50c391e3aa08c5c5e72/C is initiating minor compaction (all files) 2023-04-15 02:55:42,536 INFO [RS:0;jenkins-hbase20:34681-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of a29fa882a9e3f50c391e3aa08c5c5e72/C in TestAcidGuarantees,,1681527331158.a29fa882a9e3f50c391e3aa08c5c5e72. 2023-04-15 02:55:42,536 INFO [RS:0;jenkins-hbase20:34681-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/a29fa882a9e3f50c391e3aa08c5c5e72/C/ad509ed0bc7e4ae5b4d80a55856ce60f, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/a29fa882a9e3f50c391e3aa08c5c5e72/C/8707dafd31ff48eb9c84d4e178e25eda, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/a29fa882a9e3f50c391e3aa08c5c5e72/C/1290cf8f5e884c469991519a85d516b9] into tmpdir=hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/a29fa882a9e3f50c391e3aa08c5c5e72/.tmp, totalSize=33.3 K 2023-04-15 02:55:42,537 DEBUG [RS:0;jenkins-hbase20:34681-shortCompactions-0] compactions.Compactor(207): Compacting ad509ed0bc7e4ae5b4d80a55856ce60f, keycount=150, bloomtype=ROW, size=12.1 K, encoding=NONE, compression=NONE, seqNum=191, earliestPutTs=1681527337970 2023-04-15 02:55:42,537 DEBUG [RS:0;jenkins-hbase20:34681-shortCompactions-0] compactions.Compactor(207): Compacting 8707dafd31ff48eb9c84d4e178e25eda, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=210, earliestPutTs=1681527339129 2023-04-15 02:55:42,537 DEBUG [RS:0;jenkins-hbase20:34681-shortCompactions-0] compactions.Compactor(207): Compacting 1290cf8f5e884c469991519a85d516b9, keycount=100, bloomtype=ROW, size=9.4 K, encoding=NONE, compression=NONE, seqNum=222, earliestPutTs=1681527341259 2023-04-15 02:55:42,540 INFO [RS:0;jenkins-hbase20:34681-shortCompactions-0] throttle.PressureAwareThroughputController(145): a29fa882a9e3f50c391e3aa08c5c5e72#C#compaction#389 average throughput is unlimited, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-15 02:55:42,928 DEBUG [RS:0;jenkins-hbase20:34681-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/a29fa882a9e3f50c391e3aa08c5c5e72/.tmp/B/0173ef12b1354efb842e1ce4695fc4cd as hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/a29fa882a9e3f50c391e3aa08c5c5e72/B/0173ef12b1354efb842e1ce4695fc4cd 2023-04-15 02:55:42,931 INFO [RS:0;jenkins-hbase20:34681-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in a29fa882a9e3f50c391e3aa08c5c5e72/B of a29fa882a9e3f50c391e3aa08c5c5e72 into 0173ef12b1354efb842e1ce4695fc4cd(size=12.2 K), total size for store is 12.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-15 02:55:42,931 DEBUG [RS:0;jenkins-hbase20:34681-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for a29fa882a9e3f50c391e3aa08c5c5e72: 2023-04-15 02:55:42,931 INFO [RS:0;jenkins-hbase20:34681-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681527331158.a29fa882a9e3f50c391e3aa08c5c5e72., storeName=a29fa882a9e3f50c391e3aa08c5c5e72/B, priority=13, startTime=1681527342513; duration=0sec 2023-04-15 02:55:42,931 DEBUG [RS:0;jenkins-hbase20:34681-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-15 02:55:42,935 INFO [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=258 (bloomFilter=true), to=hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/a29fa882a9e3f50c391e3aa08c5c5e72/.tmp/A/2791a854894341ac90b837693a2a8125 2023-04-15 02:55:42,951 INFO [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=258 (bloomFilter=true), to=hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/a29fa882a9e3f50c391e3aa08c5c5e72/.tmp/B/13cb677837cb482b924e6abbb42c6a41 2023-04-15 02:55:42,954 DEBUG [RS:0;jenkins-hbase20:34681-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/a29fa882a9e3f50c391e3aa08c5c5e72/.tmp/C/258d4421a1fe497f89a4071b7edd4f2a as hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/a29fa882a9e3f50c391e3aa08c5c5e72/C/258d4421a1fe497f89a4071b7edd4f2a 2023-04-15 02:55:42,959 INFO [RS:0;jenkins-hbase20:34681-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in a29fa882a9e3f50c391e3aa08c5c5e72/C of a29fa882a9e3f50c391e3aa08c5c5e72 into 258d4421a1fe497f89a4071b7edd4f2a(size=12.2 K), total size for store is 12.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-15 02:55:42,959 DEBUG [RS:0;jenkins-hbase20:34681-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for a29fa882a9e3f50c391e3aa08c5c5e72: 2023-04-15 02:55:42,959 INFO [RS:0;jenkins-hbase20:34681-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681527331158.a29fa882a9e3f50c391e3aa08c5c5e72., storeName=a29fa882a9e3f50c391e3aa08c5c5e72/C, priority=13, startTime=1681527342513; duration=0sec 2023-04-15 02:55:42,959 DEBUG [RS:0;jenkins-hbase20:34681-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-15 02:55:43,361 INFO [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=258 (bloomFilter=true), to=hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/a29fa882a9e3f50c391e3aa08c5c5e72/.tmp/C/f7bec9a155c544618f64cf938f34ced0 2023-04-15 02:55:43,365 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/a29fa882a9e3f50c391e3aa08c5c5e72/.tmp/A/2791a854894341ac90b837693a2a8125 as hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/a29fa882a9e3f50c391e3aa08c5c5e72/A/2791a854894341ac90b837693a2a8125 2023-04-15 02:55:43,367 INFO [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/a29fa882a9e3f50c391e3aa08c5c5e72/A/2791a854894341ac90b837693a2a8125, entries=150, sequenceid=258, filesize=11.8 K 2023-04-15 02:55:43,368 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/a29fa882a9e3f50c391e3aa08c5c5e72/.tmp/B/13cb677837cb482b924e6abbb42c6a41 as hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/a29fa882a9e3f50c391e3aa08c5c5e72/B/13cb677837cb482b924e6abbb42c6a41 2023-04-15 02:55:43,370 INFO [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/a29fa882a9e3f50c391e3aa08c5c5e72/B/13cb677837cb482b924e6abbb42c6a41, entries=150, sequenceid=258, filesize=11.8 K 2023-04-15 02:55:43,371 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/a29fa882a9e3f50c391e3aa08c5c5e72/.tmp/C/f7bec9a155c544618f64cf938f34ced0 as hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/a29fa882a9e3f50c391e3aa08c5c5e72/C/f7bec9a155c544618f64cf938f34ced0 2023-04-15 02:55:43,373 INFO [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/a29fa882a9e3f50c391e3aa08c5c5e72/C/f7bec9a155c544618f64cf938f34ced0, entries=150, sequenceid=258, filesize=11.8 K 2023-04-15 02:55:43,374 INFO [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~134.18 KB/137400, heapSize ~380.25 KB/389376, currentSize=0 B/0 for a29fa882a9e3f50c391e3aa08c5c5e72 in 850ms, sequenceid=258, compaction requested=false 2023-04-15 02:55:43,374 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for a29fa882a9e3f50c391e3aa08c5c5e72: 2023-04-15 02:55:43,374 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681527331158.a29fa882a9e3f50c391e3aa08c5c5e72. 2023-04-15 02:55:43,374 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-15 02:55:43,374 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-15 02:55:43,374 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:43,374 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-15 02:55:43,374 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34681,1681527209133' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-15 02:55:43,375 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:43,375 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:43,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:43,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:55:43,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:55:43,375 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:43,375 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-15 02:55:43,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:55:43,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:55:43,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:55:43,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:43,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:55:43,377 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34681,1681527209133' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-15 02:55:43,377 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-15 02:55:43,377 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@46e0a01[Count = 0] remaining members to acquire global barrier 2023-04-15 02:55:43,377 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:43,377 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:43,377 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:43,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:43,377 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-15 02:55:43,377 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-15 02:55:43,377 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34681,1681527209133' in zk 2023-04-15 02:55:43,378 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:43,378 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-15 02:55:43,378 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-15 02:55:43,378 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:43,378 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-15 02:55:43,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:43,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:55:43,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:55:43,378 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-15 02:55:43,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:55:43,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:55:43,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:55:43,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:43,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:55:43,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:55:43,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:43,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34681,1681527209133': 2023-04-15 02:55:43,380 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34681,1681527209133' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-15 02:55:43,380 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-15 02:55:43,380 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-15 02:55:43,380 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-15 02:55:43,380 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-15 02:55:43,380 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-15 02:55:43,381 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:43,381 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-15 02:55:43,381 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:43,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:43,381 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-15 02:55:43,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-15 02:55:43,381 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:43,381 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:43,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:43,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:55:43,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:43,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:55:43,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:55:43,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:55:43,382 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:43,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:55:43,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:55:43,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:43,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:55:43,383 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-15 02:55:43,383 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-15 02:55:43,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-15 02:55:43,384 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-15 02:55:43,384 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-15 02:55:43,383 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-15 02:55:43,384 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-15 02:55:43,384 DEBUG [Thread-2739] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-15 02:55:43,384 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-15 02:55:43,384 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-15 02:55:43,384 DEBUG [Thread-2739] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-15 02:55:43,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-15 02:55:43,384 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:43,384 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:55:43,384 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:55:43,384 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:43,384 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:43,385 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:43,385 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:43,462 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34681] regionserver.HRegion(9158): Flush requested on a29fa882a9e3f50c391e3aa08c5c5e72 2023-04-15 02:55:43,462 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing a29fa882a9e3f50c391e3aa08c5c5e72 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-15 02:55:43,462 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a29fa882a9e3f50c391e3aa08c5c5e72, store=A 2023-04-15 02:55:43,462 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-15 02:55:43,462 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a29fa882a9e3f50c391e3aa08c5c5e72, store=B 2023-04-15 02:55:43,462 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-15 02:55:43,463 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a29fa882a9e3f50c391e3aa08c5c5e72, store=C 2023-04-15 02:55:43,463 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-15 02:55:43,484 DEBUG [Thread-2739] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-15 02:55:43,485 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-15 02:55:43,486 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-15 02:55:43,486 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-15 02:55:43,487 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-15 02:55:43,487 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-15 02:55:43,487 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-15 02:55:43,487 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-15 02:55:43,487 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-15 02:55:43,487 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:43,487 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:55:43,488 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:43,488 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-15 02:55:43,488 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-15 02:55:43,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-15 02:55:43,488 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:43,488 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-15 02:55:43,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:55:43,489 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:43,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-15 02:55:43,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:55:43,489 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-15 02:55:43,489 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-15 02:55:43,489 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-15 02:55:43,489 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-15 02:55:43,489 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-15 02:55:43,489 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-15 02:55:43,489 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-15 02:55:43,489 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-15 02:55:43,489 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681527331158.a29fa882a9e3f50c391e3aa08c5c5e72. 2023-04-15 02:55:43,489 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681527331158.a29fa882a9e3f50c391e3aa08c5c5e72. started... 2023-04-15 02:55:43,489 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681527331158.a29fa882a9e3f50c391e3aa08c5c5e72. as already flushing 2023-04-15 02:55:43,489 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681527331158.a29fa882a9e3f50c391e3aa08c5c5e72. 2023-04-15 02:55:43,490 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-15 02:55:43,490 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-15 02:55:43,490 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:43,490 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-15 02:55:43,490 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34681,1681527209133' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-15 02:55:43,490 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:43,490 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:43,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:43,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:55:43,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:55:43,491 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:43,491 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-15 02:55:43,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:55:43,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:55:43,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:55:43,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:43,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:55:43,492 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34681,1681527209133' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-15 02:55:43,492 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-15 02:55:43,492 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@278929b[Count = 0] remaining members to acquire global barrier 2023-04-15 02:55:43,492 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:43,492 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:43,492 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:43,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:43,492 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-15 02:55:43,492 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-15 02:55:43,492 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34681,1681527209133' in zk 2023-04-15 02:55:43,492 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:43,492 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-15 02:55:43,493 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-15 02:55:43,493 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-15 02:55:43,493 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:43,493 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-15 02:55:43,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:43,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:55:43,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:55:43,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:55:43,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:55:43,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:55:43,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:43,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:55:43,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:55:43,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:43,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34681,1681527209133': 2023-04-15 02:55:43,494 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34681,1681527209133' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-15 02:55:43,494 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-15 02:55:43,495 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-15 02:55:43,495 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-15 02:55:43,495 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-15 02:55:43,495 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-15 02:55:43,496 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:43,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:43,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:55:43,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:55:43,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:55:43,496 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:43,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:55:43,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:55:43,497 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1681527343470, max timestamp=1681527343495Num uniques 859; 2023-04-15 02:55:43,497 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1681527343470, max timestamp=1681527343495Num uniques 859; 2023-04-15 02:55:43,497 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-3] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-15 02:55:43,497 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-5] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-15 02:55:43,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:55:43,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:43,497 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:43,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:55:43,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:55:43,498 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1681527343470, max timestamp=1681527343495Num uniques 859; 2023-04-15 02:55:43,498 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-4] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-15 02:55:43,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:43,498 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:43,498 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-15 02:55:43,498 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:43,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:43,498 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-15 02:55:43,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-15 02:55:43,499 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:43,499 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-15 02:55:43,499 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:55:43,499 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-15 02:55:43,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-15 02:55:43,499 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-15 02:55:43,499 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-15 02:55:43,499 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:55:43,499 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-15 02:55:43,499 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:43,500 DEBUG [Thread-2739] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-15 02:55:43,500 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:43,500 DEBUG [Thread-2739] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-15 02:55:43,500 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:43,500 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:43,545 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a29fa882a9e3f50c391e3aa08c5c5e72, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:55:43,545 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34681] ipc.CallRunner(144): callId: 133 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:40234 deadline: 1681527403537, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a29fa882a9e3f50c391e3aa08c5c5e72, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:43,545 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a29fa882a9e3f50c391e3aa08c5c5e72, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:55:43,545 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34681] ipc.CallRunner(144): callId: 145 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:40210 deadline: 1681527403537, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a29fa882a9e3f50c391e3aa08c5c5e72, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:43,551 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a29fa882a9e3f50c391e3aa08c5c5e72, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:55:43,551 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34681] ipc.CallRunner(144): callId: 145 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:40192 deadline: 1681527403545, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a29fa882a9e3f50c391e3aa08c5c5e72, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:43,600 DEBUG [Thread-2739] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-15 02:55:43,600 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-15 02:55:43,601 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-15 02:55:43,602 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-15 02:55:43,602 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-15 02:55:43,602 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-15 02:55:43,602 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-15 02:55:43,602 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-15 02:55:43,602 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-15 02:55:43,602 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:43,602 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:55:43,603 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-15 02:55:43,603 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:43,603 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-15 02:55:43,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-15 02:55:43,603 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:43,603 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-15 02:55:43,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:55:43,604 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:43,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-15 02:55:43,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:55:43,604 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-15 02:55:43,604 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-15 02:55:43,604 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-15 02:55:43,604 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-15 02:55:43,604 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-15 02:55:43,604 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-15 02:55:43,604 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-15 02:55:43,605 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-15 02:55:43,605 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681527331158.a29fa882a9e3f50c391e3aa08c5c5e72. 2023-04-15 02:55:43,605 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681527331158.a29fa882a9e3f50c391e3aa08c5c5e72. started... 2023-04-15 02:55:43,605 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681527331158.a29fa882a9e3f50c391e3aa08c5c5e72. as already flushing 2023-04-15 02:55:43,605 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681527331158.a29fa882a9e3f50c391e3aa08c5c5e72. 2023-04-15 02:55:43,605 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-15 02:55:43,605 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-15 02:55:43,605 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:43,605 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-15 02:55:43,605 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34681,1681527209133' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-15 02:55:43,605 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:43,605 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:43,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:43,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:55:43,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:55:43,606 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:43,606 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-15 02:55:43,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:55:43,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:55:43,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:55:43,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:43,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:55:43,607 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34681,1681527209133' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-15 02:55:43,607 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@65c27f2f[Count = 0] remaining members to acquire global barrier 2023-04-15 02:55:43,607 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-15 02:55:43,607 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:43,607 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:43,607 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:43,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:43,607 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:43,607 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-15 02:55:43,607 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-15 02:55:43,607 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-15 02:55:43,607 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34681,1681527209133' in zk 2023-04-15 02:55:43,608 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:43,608 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-15 02:55:43,608 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-15 02:55:43,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:43,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:55:43,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:55:43,608 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-15 02:55:43,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:55:43,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:55:43,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:55:43,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:43,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:55:43,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:55:43,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:43,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34681,1681527209133': 2023-04-15 02:55:43,610 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34681,1681527209133' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-15 02:55:43,610 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-15 02:55:43,610 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-15 02:55:43,610 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-15 02:55:43,610 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-15 02:55:43,610 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-15 02:55:43,611 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:43,611 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:43,611 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-15 02:55:43,611 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:43,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:43,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:43,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:55:43,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:55:43,611 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-15 02:55:43,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-15 02:55:43,611 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:43,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:55:43,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:43,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:55:43,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:55:43,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:55:43,612 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:43,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:43,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:55:43,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:55:43,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:43,613 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:43,614 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:55:43,614 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:55:43,614 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:43,614 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-15 02:55:43,614 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-15 02:55:43,614 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-15 02:55:43,614 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-15 02:55:43,614 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-15 02:55:43,614 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:43,614 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-15 02:55:43,614 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:43,614 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:43,614 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-15 02:55:43,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-15 02:55:43,614 DEBUG [Thread-2739] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-15 02:55:43,614 DEBUG [Thread-2739] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-15 02:55:43,614 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-15 02:55:43,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-15 02:55:43,650 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a29fa882a9e3f50c391e3aa08c5c5e72, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:55:43,650 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34681] ipc.CallRunner(144): callId: 147 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:40210 deadline: 1681527403646, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a29fa882a9e3f50c391e3aa08c5c5e72, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:43,651 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a29fa882a9e3f50c391e3aa08c5c5e72, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:55:43,651 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34681] ipc.CallRunner(144): callId: 135 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:40234 deadline: 1681527403646, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a29fa882a9e3f50c391e3aa08c5c5e72, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:43,660 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a29fa882a9e3f50c391e3aa08c5c5e72, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:55:43,661 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34681] ipc.CallRunner(144): callId: 147 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:40192 deadline: 1681527403652, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a29fa882a9e3f50c391e3aa08c5c5e72, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:43,714 DEBUG [Thread-2739] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-15 02:55:43,715 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-15 02:55:43,716 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-15 02:55:43,716 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-15 02:55:43,717 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-15 02:55:43,717 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-15 02:55:43,717 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-15 02:55:43,717 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-15 02:55:43,717 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-15 02:55:43,717 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:43,717 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:55:43,718 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:43,718 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-15 02:55:43,718 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-15 02:55:43,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-15 02:55:43,718 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:43,718 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-15 02:55:43,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:55:43,718 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:43,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-15 02:55:43,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:55:43,719 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-15 02:55:43,719 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-15 02:55:43,719 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-15 02:55:43,719 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-15 02:55:43,719 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-15 02:55:43,719 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-15 02:55:43,719 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-15 02:55:43,719 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-15 02:55:43,719 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681527331158.a29fa882a9e3f50c391e3aa08c5c5e72. 2023-04-15 02:55:43,719 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681527331158.a29fa882a9e3f50c391e3aa08c5c5e72. started... 2023-04-15 02:55:43,719 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681527331158.a29fa882a9e3f50c391e3aa08c5c5e72. as already flushing 2023-04-15 02:55:43,719 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681527331158.a29fa882a9e3f50c391e3aa08c5c5e72. 2023-04-15 02:55:43,719 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-15 02:55:43,719 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-15 02:55:43,719 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:43,719 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-15 02:55:43,719 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34681,1681527209133' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-15 02:55:43,726 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:43,726 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:43,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:43,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:55:43,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:55:43,727 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:43,727 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-15 02:55:43,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:55:43,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:55:43,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:55:43,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:43,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:55:43,728 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34681,1681527209133' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-15 02:55:43,728 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7467c271[Count = 0] remaining members to acquire global barrier 2023-04-15 02:55:43,728 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-15 02:55:43,728 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:43,728 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:43,728 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:43,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:43,728 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-15 02:55:43,728 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:43,728 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-15 02:55:43,728 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-15 02:55:43,728 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34681,1681527209133' in zk 2023-04-15 02:55:43,730 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:43,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:43,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:55:43,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:55:43,730 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-15 02:55:43,730 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-15 02:55:43,730 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-15 02:55:43,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:55:43,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:55:43,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:55:43,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:43,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:55:43,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:55:43,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:43,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34681,1681527209133': 2023-04-15 02:55:43,731 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34681,1681527209133' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-15 02:55:43,731 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-15 02:55:43,731 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-15 02:55:43,731 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-15 02:55:43,731 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-15 02:55:43,731 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-15 02:55:43,732 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:43,732 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:43,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:43,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:55:43,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:55:43,732 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:43,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:43,732 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-15 02:55:43,732 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:43,732 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-15 02:55:43,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-15 02:55:43,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:55:43,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:43,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:55:43,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:55:43,733 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:43,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:55:43,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:43,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:55:43,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:55:43,735 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:43,735 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:55:43,735 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:55:43,735 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-15 02:55:43,735 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:43,735 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-15 02:55:43,735 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-15 02:55:43,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-15 02:55:43,735 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-15 02:55:43,735 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-15 02:55:43,735 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:43,735 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:43,735 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:43,735 DEBUG [Thread-2739] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-15 02:55:43,735 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-15 02:55:43,735 DEBUG [Thread-2739] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-15 02:55:43,735 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-15 02:55:43,735 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-15 02:55:43,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-15 02:55:43,835 DEBUG [Thread-2739] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-15 02:55:43,836 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-15 02:55:43,837 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-15 02:55:43,838 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-15 02:55:43,838 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-15 02:55:43,838 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-15 02:55:43,838 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-15 02:55:43,838 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-15 02:55:43,838 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-15 02:55:43,838 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:43,838 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:55:43,839 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-15 02:55:43,839 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:43,839 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-15 02:55:43,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-15 02:55:43,840 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:43,840 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-15 02:55:43,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:55:43,840 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:43,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-15 02:55:43,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:55:43,840 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-15 02:55:43,841 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-15 02:55:43,841 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-15 02:55:43,841 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-15 02:55:43,841 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-15 02:55:43,841 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-15 02:55:43,841 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-15 02:55:43,841 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-15 02:55:43,841 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681527331158.a29fa882a9e3f50c391e3aa08c5c5e72. 2023-04-15 02:55:43,841 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681527331158.a29fa882a9e3f50c391e3aa08c5c5e72. started... 2023-04-15 02:55:43,841 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681527331158.a29fa882a9e3f50c391e3aa08c5c5e72. as already flushing 2023-04-15 02:55:43,841 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681527331158.a29fa882a9e3f50c391e3aa08c5c5e72. 2023-04-15 02:55:43,841 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-15 02:55:43,841 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-15 02:55:43,841 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:43,841 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-15 02:55:43,841 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34681,1681527209133' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-15 02:55:43,842 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:43,842 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:43,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:43,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:55:43,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:55:43,843 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:43,843 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-15 02:55:43,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:55:43,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:55:43,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:55:43,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:43,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:55:43,844 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34681,1681527209133' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-15 02:55:43,844 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-15 02:55:43,844 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2de0fe[Count = 0] remaining members to acquire global barrier 2023-04-15 02:55:43,844 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:43,844 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:43,844 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:43,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:43,844 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-15 02:55:43,844 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-15 02:55:43,844 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:43,844 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-15 02:55:43,844 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34681,1681527209133' in zk 2023-04-15 02:55:43,845 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:43,845 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-15 02:55:43,845 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-15 02:55:43,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:43,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:55:43,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:55:43,845 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-15 02:55:43,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:55:43,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:55:43,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:55:43,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:43,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:55:43,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:55:43,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:43,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34681,1681527209133': 2023-04-15 02:55:43,847 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34681,1681527209133' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-15 02:55:43,847 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-15 02:55:43,847 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-15 02:55:43,847 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-15 02:55:43,847 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-15 02:55:43,847 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-15 02:55:43,848 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:43,848 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:43,848 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:43,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:43,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:55:43,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:55:43,848 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-15 02:55:43,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:43,849 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-15 02:55:43,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-15 02:55:43,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:55:43,849 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:43,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:43,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:55:43,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:55:43,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:55:43,850 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:43,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:43,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:55:43,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:55:43,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:43,852 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:43,852 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:55:43,852 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-15 02:55:43,852 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:55:43,852 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-15 02:55:43,852 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-15 02:55:43,852 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-15 02:55:43,852 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-15 02:55:43,852 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-15 02:55:43,852 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:43,852 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-15 02:55:43,852 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:43,852 DEBUG [Thread-2739] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-15 02:55:43,852 DEBUG [Thread-2739] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-15 02:55:43,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-15 02:55:43,852 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:43,852 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:43,853 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-15 02:55:43,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-15 02:55:43,856 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a29fa882a9e3f50c391e3aa08c5c5e72, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:55:43,856 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34681] ipc.CallRunner(144): callId: 137 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:40234 deadline: 1681527403852, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a29fa882a9e3f50c391e3aa08c5c5e72, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:43,857 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a29fa882a9e3f50c391e3aa08c5c5e72, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:55:43,857 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34681] ipc.CallRunner(144): callId: 149 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:40210 deadline: 1681527403853, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a29fa882a9e3f50c391e3aa08c5c5e72, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:43,862 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a29fa882a9e3f50c391e3aa08c5c5e72, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:55:43,863 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34681] ipc.CallRunner(144): callId: 149 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:40192 deadline: 1681527403861, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a29fa882a9e3f50c391e3aa08c5c5e72, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:43,868 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=271 (bloomFilter=true), to=hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/a29fa882a9e3f50c391e3aa08c5c5e72/.tmp/A/6637bb1f67c841e3a1430bf06afb4102 2023-04-15 02:55:43,952 DEBUG [Thread-2739] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-15 02:55:43,953 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-15 02:55:43,954 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-15 02:55:43,955 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-15 02:55:43,955 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-15 02:55:43,955 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-15 02:55:43,955 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-15 02:55:43,955 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-15 02:55:43,955 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-15 02:55:43,955 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:43,955 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:55:43,957 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-15 02:55:43,957 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-15 02:55:43,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-15 02:55:43,957 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:43,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:55:43,957 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:43,957 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-15 02:55:43,957 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:43,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-15 02:55:43,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:55:43,958 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-15 02:55:43,958 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-15 02:55:43,958 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-15 02:55:43,958 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-15 02:55:43,958 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-15 02:55:43,958 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-15 02:55:43,958 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-15 02:55:43,958 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-15 02:55:43,958 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681527331158.a29fa882a9e3f50c391e3aa08c5c5e72. 2023-04-15 02:55:43,958 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681527331158.a29fa882a9e3f50c391e3aa08c5c5e72. started... 2023-04-15 02:55:43,958 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681527331158.a29fa882a9e3f50c391e3aa08c5c5e72. as already flushing 2023-04-15 02:55:43,959 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681527331158.a29fa882a9e3f50c391e3aa08c5c5e72. 2023-04-15 02:55:43,959 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-15 02:55:43,959 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-15 02:55:43,959 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:43,959 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-15 02:55:43,959 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34681,1681527209133' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-15 02:55:43,960 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:43,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:43,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:55:43,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:55:43,960 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:43,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:55:43,960 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:43,960 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-15 02:55:43,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:55:43,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:55:43,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:43,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:55:43,961 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34681,1681527209133' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-15 02:55:43,961 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4607562a[Count = 0] remaining members to acquire global barrier 2023-04-15 02:55:43,961 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-15 02:55:43,961 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:43,961 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:43,961 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:43,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:43,961 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-15 02:55:43,962 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-15 02:55:43,962 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:43,962 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-15 02:55:43,962 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34681,1681527209133' in zk 2023-04-15 02:55:43,963 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:43,963 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-15 02:55:43,963 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-15 02:55:43,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:43,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:55:43,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:55:43,963 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-15 02:55:43,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:55:43,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:55:43,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:55:43,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:43,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:55:43,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:55:43,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:43,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34681,1681527209133': 2023-04-15 02:55:43,964 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34681,1681527209133' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-15 02:55:43,964 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-15 02:55:43,964 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-15 02:55:43,964 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-15 02:55:43,965 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-15 02:55:43,965 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-15 02:55:43,965 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:43,965 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:43,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:43,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:55:43,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:55:43,965 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-15 02:55:43,965 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:43,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:43,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:55:43,966 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-15 02:55:43,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-15 02:55:43,966 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:43,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:43,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:55:43,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:55:43,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:55:43,967 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:43,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:43,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:55:43,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:55:43,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:43,968 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:43,968 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:55:43,968 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:55:43,968 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:43,968 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:43,968 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-15 02:55:43,968 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:43,968 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-15 02:55:43,968 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-15 02:55:43,968 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:43,968 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-15 02:55:43,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-15 02:55:43,968 DEBUG [Thread-2739] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-15 02:55:43,968 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-15 02:55:43,968 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-15 02:55:43,968 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-15 02:55:43,968 DEBUG [Thread-2739] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-15 02:55:43,969 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-15 02:55:43,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-15 02:55:44,069 DEBUG [Thread-2739] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-15 02:55:44,069 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-15 02:55:44,070 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-15 02:55:44,071 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-15 02:55:44,071 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-15 02:55:44,071 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-15 02:55:44,071 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-15 02:55:44,072 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-15 02:55:44,072 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-15 02:55:44,072 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:44,072 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:55:44,073 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-15 02:55:44,073 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:44,073 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-15 02:55:44,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-15 02:55:44,073 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:44,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:55:44,073 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-15 02:55:44,073 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:44,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-15 02:55:44,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:55:44,073 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-15 02:55:44,073 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-15 02:55:44,074 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-15 02:55:44,074 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-15 02:55:44,074 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-15 02:55:44,074 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-15 02:55:44,074 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-15 02:55:44,074 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-15 02:55:44,074 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681527331158.a29fa882a9e3f50c391e3aa08c5c5e72. 2023-04-15 02:55:44,074 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681527331158.a29fa882a9e3f50c391e3aa08c5c5e72. started... 2023-04-15 02:55:44,074 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681527331158.a29fa882a9e3f50c391e3aa08c5c5e72. as already flushing 2023-04-15 02:55:44,074 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681527331158.a29fa882a9e3f50c391e3aa08c5c5e72. 2023-04-15 02:55:44,074 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-15 02:55:44,074 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-15 02:55:44,074 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:44,074 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-15 02:55:44,074 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34681,1681527209133' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-15 02:55:44,075 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:44,075 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:44,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:44,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:55:44,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:55:44,075 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:44,075 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-15 02:55:44,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:55:44,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:55:44,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:55:44,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:44,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:55:44,076 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34681,1681527209133' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-15 02:55:44,076 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-15 02:55:44,076 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@30ca62c2[Count = 0] remaining members to acquire global barrier 2023-04-15 02:55:44,076 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:44,077 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:44,077 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:44,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:44,077 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-15 02:55:44,077 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-15 02:55:44,077 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:44,077 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-15 02:55:44,077 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34681,1681527209133' in zk 2023-04-15 02:55:44,078 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-15 02:55:44,078 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:44,078 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-15 02:55:44,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:44,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:55:44,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:55:44,078 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-15 02:55:44,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:55:44,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:55:44,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:55:44,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:44,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:55:44,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:55:44,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:44,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34681,1681527209133': 2023-04-15 02:55:44,080 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34681,1681527209133' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-15 02:55:44,080 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-15 02:55:44,080 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-15 02:55:44,080 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-15 02:55:44,080 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-15 02:55:44,080 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-15 02:55:44,081 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:44,081 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:44,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:44,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:55:44,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:55:44,081 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-15 02:55:44,081 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:44,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:44,081 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-15 02:55:44,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:55:44,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-15 02:55:44,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:44,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:55:44,081 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:44,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:55:44,082 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:44,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:55:44,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:44,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:55:44,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:55:44,084 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:44,084 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:55:44,084 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:55:44,084 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-15 02:55:44,084 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:44,084 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:44,084 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-15 02:55:44,084 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-15 02:55:44,084 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-15 02:55:44,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-15 02:55:44,084 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-15 02:55:44,084 DEBUG [Thread-2739] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-15 02:55:44,084 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-15 02:55:44,084 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:44,084 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-15 02:55:44,084 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:44,084 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-15 02:55:44,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-15 02:55:44,084 DEBUG [Thread-2739] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-15 02:55:44,162 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a29fa882a9e3f50c391e3aa08c5c5e72, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:55:44,162 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34681] ipc.CallRunner(144): callId: 139 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:40234 deadline: 1681527404158, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a29fa882a9e3f50c391e3aa08c5c5e72, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:44,162 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a29fa882a9e3f50c391e3aa08c5c5e72, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:55:44,162 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34681] ipc.CallRunner(144): callId: 151 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:40210 deadline: 1681527404159, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a29fa882a9e3f50c391e3aa08c5c5e72, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:44,168 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a29fa882a9e3f50c391e3aa08c5c5e72, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:55:44,168 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34681] ipc.CallRunner(144): callId: 151 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:40192 deadline: 1681527404165, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a29fa882a9e3f50c391e3aa08c5c5e72, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:44,184 DEBUG [Thread-2739] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-15 02:55:44,185 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-15 02:55:44,186 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-15 02:55:44,187 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-15 02:55:44,187 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-15 02:55:44,187 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-15 02:55:44,187 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-15 02:55:44,187 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-15 02:55:44,187 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-15 02:55:44,187 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:44,188 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:55:44,188 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:44,188 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-15 02:55:44,188 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-15 02:55:44,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-15 02:55:44,188 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:44,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:55:44,188 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-15 02:55:44,189 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:44,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-15 02:55:44,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:55:44,189 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-15 02:55:44,189 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-15 02:55:44,189 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-15 02:55:44,189 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-15 02:55:44,189 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-15 02:55:44,189 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-15 02:55:44,189 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-15 02:55:44,190 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-15 02:55:44,190 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681527331158.a29fa882a9e3f50c391e3aa08c5c5e72. 2023-04-15 02:55:44,190 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681527331158.a29fa882a9e3f50c391e3aa08c5c5e72. started... 2023-04-15 02:55:44,190 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681527331158.a29fa882a9e3f50c391e3aa08c5c5e72. as already flushing 2023-04-15 02:55:44,190 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681527331158.a29fa882a9e3f50c391e3aa08c5c5e72. 2023-04-15 02:55:44,190 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-15 02:55:44,190 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-15 02:55:44,190 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:44,190 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-15 02:55:44,190 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34681,1681527209133' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-15 02:55:44,191 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:44,191 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:44,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:44,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:55:44,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:55:44,191 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:44,191 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-15 02:55:44,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:55:44,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:55:44,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:55:44,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:44,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:55:44,192 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34681,1681527209133' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-15 02:55:44,192 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-15 02:55:44,192 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7a744ae0[Count = 0] remaining members to acquire global barrier 2023-04-15 02:55:44,192 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:44,192 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:44,192 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:44,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:44,193 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-15 02:55:44,193 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-15 02:55:44,193 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:44,193 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-15 02:55:44,193 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34681,1681527209133' in zk 2023-04-15 02:55:44,193 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-15 02:55:44,193 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-15 02:55:44,193 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-15 02:55:44,194 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:44,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:44,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:55:44,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:55:44,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:55:44,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:55:44,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:55:44,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:44,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:55:44,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:55:44,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:44,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34681,1681527209133': 2023-04-15 02:55:44,195 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34681,1681527209133' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-15 02:55:44,195 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-15 02:55:44,195 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-15 02:55:44,196 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-15 02:55:44,196 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-15 02:55:44,196 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-15 02:55:44,196 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:44,196 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:44,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:44,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:55:44,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:55:44,196 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:44,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:44,196 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-15 02:55:44,197 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:44,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:55:44,197 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-15 02:55:44,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-15 02:55:44,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:55:44,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:44,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:55:44,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:55:44,197 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:44,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:44,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:55:44,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:55:44,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:44,199 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-15 02:55:44,199 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-15 02:55:44,199 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-15 02:55:44,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-15 02:55:44,199 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-15 02:55:44,199 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-15 02:55:44,199 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-15 02:55:44,199 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-15 02:55:44,199 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:44,199 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:55:44,199 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:55:44,199 DEBUG [Thread-2739] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-15 02:55:44,199 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-15 02:55:44,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-15 02:55:44,199 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:44,199 DEBUG [Thread-2739] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-15 02:55:44,199 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:44,200 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:44,200 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:44,226 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a29fa882a9e3f50c391e3aa08c5c5e72, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:55:44,227 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34681] ipc.CallRunner(144): callId: 70 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:40212 deadline: 1681527404223, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a29fa882a9e3f50c391e3aa08c5c5e72, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:44,228 DEBUG [Thread-2733] client.RpcRetryingCallerImpl(129): Call exception, tries=7, retries=16, started=8201 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a29fa882a9e3f50c391e3aa08c5c5e72, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_2' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1681527331158.a29fa882a9e3f50c391e3aa08c5c5e72., hostname=jenkins-hbase20.apache.org,34681,1681527209133, seqNum=2, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a29fa882a9e3f50c391e3aa08c5c5e72, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor31.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a29fa882a9e3f50c391e3aa08c5c5e72, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-15 02:55:44,233 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a29fa882a9e3f50c391e3aa08c5c5e72, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:55:44,233 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34681] ipc.CallRunner(144): callId: 69 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:40158 deadline: 1681527404230, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a29fa882a9e3f50c391e3aa08c5c5e72, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:44,235 DEBUG [Thread-2731] client.RpcRetryingCallerImpl(129): Call exception, tries=7, retries=16, started=8208 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a29fa882a9e3f50c391e3aa08c5c5e72, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_1' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1681527331158.a29fa882a9e3f50c391e3aa08c5c5e72., hostname=jenkins-hbase20.apache.org,34681,1681527209133, seqNum=2, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a29fa882a9e3f50c391e3aa08c5c5e72, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor31.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a29fa882a9e3f50c391e3aa08c5c5e72, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-15 02:55:44,275 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=271 (bloomFilter=true), to=hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/a29fa882a9e3f50c391e3aa08c5c5e72/.tmp/B/6300d8f8bbcd462081165d7f3228ba43 2023-04-15 02:55:44,300 DEBUG [Thread-2739] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-15 02:55:44,300 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-15 02:55:44,301 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-15 02:55:44,302 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-15 02:55:44,302 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-15 02:55:44,302 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-15 02:55:44,302 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-15 02:55:44,302 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-15 02:55:44,302 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-15 02:55:44,302 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:44,302 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:55:44,303 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:44,303 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-15 02:55:44,303 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-15 02:55:44,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-15 02:55:44,303 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:44,303 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-15 02:55:44,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:55:44,304 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:44,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-15 02:55:44,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:55:44,304 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-15 02:55:44,304 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-15 02:55:44,304 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-15 02:55:44,304 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-15 02:55:44,304 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-15 02:55:44,304 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-15 02:55:44,304 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-15 02:55:44,304 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681527331158.a29fa882a9e3f50c391e3aa08c5c5e72. 2023-04-15 02:55:44,304 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681527331158.a29fa882a9e3f50c391e3aa08c5c5e72. started... 2023-04-15 02:55:44,304 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681527331158.a29fa882a9e3f50c391e3aa08c5c5e72. as already flushing 2023-04-15 02:55:44,305 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681527331158.a29fa882a9e3f50c391e3aa08c5c5e72. 2023-04-15 02:55:44,305 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-15 02:55:44,305 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-15 02:55:44,305 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:44,305 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-15 02:55:44,305 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34681,1681527209133' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-15 02:55:44,305 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:44,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:44,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:55:44,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:55:44,305 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:44,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:55:44,306 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:44,306 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-15 02:55:44,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:55:44,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:55:44,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:44,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:55:44,307 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34681,1681527209133' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-15 02:55:44,307 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-15 02:55:44,307 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@15b17f40[Count = 0] remaining members to acquire global barrier 2023-04-15 02:55:44,307 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:44,308 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:44,308 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:44,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:44,308 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-15 02:55:44,308 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-15 02:55:44,308 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:44,309 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-15 02:55:44,308 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34681,1681527209133' in zk 2023-04-15 02:55:44,309 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:44,309 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-15 02:55:44,309 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-15 02:55:44,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:44,309 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-15 02:55:44,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:55:44,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:55:44,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:55:44,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:55:44,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:55:44,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:44,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:55:44,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:55:44,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:44,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34681,1681527209133': 2023-04-15 02:55:44,311 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34681,1681527209133' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-15 02:55:44,311 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-15 02:55:44,311 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-15 02:55:44,311 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-15 02:55:44,311 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-15 02:55:44,311 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-15 02:55:44,312 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:44,312 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:44,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:44,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:55:44,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:55:44,312 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-15 02:55:44,312 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:44,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:44,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:55:44,313 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:44,313 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-15 02:55:44,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-15 02:55:44,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:55:44,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:44,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:55:44,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:55:44,313 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:44,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:44,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:55:44,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:55:44,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:44,316 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-15 02:55:44,316 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:44,316 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-15 02:55:44,316 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-15 02:55:44,316 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-15 02:55:44,316 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-15 02:55:44,316 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-15 02:55:44,316 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-15 02:55:44,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-15 02:55:44,316 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:55:44,316 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:55:44,316 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:44,316 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:44,316 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:44,316 DEBUG [Thread-2739] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-15 02:55:44,316 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:44,316 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-15 02:55:44,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-15 02:55:44,317 DEBUG [Thread-2739] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-15 02:55:44,417 DEBUG [Thread-2739] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-15 02:55:44,417 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-15 02:55:44,419 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-15 02:55:44,419 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-15 02:55:44,419 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-15 02:55:44,419 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-15 02:55:44,419 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-15 02:55:44,419 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-15 02:55:44,419 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-15 02:55:44,420 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:44,420 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:55:44,421 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:44,421 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-15 02:55:44,421 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-15 02:55:44,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-15 02:55:44,421 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:44,421 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-15 02:55:44,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:55:44,421 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:44,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-15 02:55:44,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:55:44,422 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-15 02:55:44,422 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-15 02:55:44,422 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-15 02:55:44,422 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-15 02:55:44,422 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-15 02:55:44,422 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-15 02:55:44,422 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-15 02:55:44,422 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-15 02:55:44,422 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681527331158.a29fa882a9e3f50c391e3aa08c5c5e72. 2023-04-15 02:55:44,422 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681527331158.a29fa882a9e3f50c391e3aa08c5c5e72. started... 2023-04-15 02:55:44,422 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681527331158.a29fa882a9e3f50c391e3aa08c5c5e72. as already flushing 2023-04-15 02:55:44,422 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681527331158.a29fa882a9e3f50c391e3aa08c5c5e72. 2023-04-15 02:55:44,422 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-15 02:55:44,422 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-15 02:55:44,422 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:44,422 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-15 02:55:44,422 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34681,1681527209133' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-15 02:55:44,423 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:44,423 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:44,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:44,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:55:44,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:55:44,423 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:44,423 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-15 02:55:44,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:55:44,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:55:44,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:55:44,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:44,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:55:44,424 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34681,1681527209133' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-15 02:55:44,424 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-15 02:55:44,424 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@697bddda[Count = 0] remaining members to acquire global barrier 2023-04-15 02:55:44,424 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:44,425 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:44,425 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:44,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:44,425 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-15 02:55:44,425 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-15 02:55:44,425 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34681,1681527209133' in zk 2023-04-15 02:55:44,425 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:44,425 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-15 02:55:44,426 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:44,426 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-15 02:55:44,426 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-15 02:55:44,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:44,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:55:44,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:55:44,426 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-15 02:55:44,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:55:44,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:55:44,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:55:44,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:44,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:55:44,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:55:44,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:44,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34681,1681527209133': 2023-04-15 02:55:44,427 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34681,1681527209133' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-15 02:55:44,427 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-15 02:55:44,427 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-15 02:55:44,427 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-15 02:55:44,427 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-15 02:55:44,427 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-15 02:55:44,428 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:44,428 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:44,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:44,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:55:44,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:55:44,428 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-15 02:55:44,428 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:44,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:44,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:55:44,429 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:44,429 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-15 02:55:44,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:55:44,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-15 02:55:44,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:55:44,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:44,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:55:44,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:44,429 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:44,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:55:44,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:55:44,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:44,431 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:44,431 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-15 02:55:44,431 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:55:44,431 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-15 02:55:44,431 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-15 02:55:44,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-15 02:55:44,431 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-15 02:55:44,431 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-15 02:55:44,431 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-15 02:55:44,431 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:55:44,431 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:44,431 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:44,431 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:44,431 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:44,431 DEBUG [Thread-2739] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-15 02:55:44,431 DEBUG [Thread-2739] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-15 02:55:44,431 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-15 02:55:44,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-15 02:55:44,531 DEBUG [Thread-2739] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-15 02:55:44,532 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-15 02:55:44,533 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-15 02:55:44,533 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-15 02:55:44,534 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-15 02:55:44,534 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-15 02:55:44,534 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-15 02:55:44,534 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-15 02:55:44,534 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-15 02:55:44,534 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:44,534 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:55:44,535 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:44,535 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-15 02:55:44,535 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-15 02:55:44,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-15 02:55:44,535 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:44,535 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-15 02:55:44,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:55:44,535 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:44,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-15 02:55:44,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:55:44,535 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-15 02:55:44,536 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-15 02:55:44,536 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-15 02:55:44,536 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-15 02:55:44,536 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-15 02:55:44,536 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-15 02:55:44,536 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-15 02:55:44,536 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681527331158.a29fa882a9e3f50c391e3aa08c5c5e72. 2023-04-15 02:55:44,536 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681527331158.a29fa882a9e3f50c391e3aa08c5c5e72. started... 2023-04-15 02:55:44,536 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-15 02:55:44,536 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681527331158.a29fa882a9e3f50c391e3aa08c5c5e72. as already flushing 2023-04-15 02:55:44,536 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681527331158.a29fa882a9e3f50c391e3aa08c5c5e72. 2023-04-15 02:55:44,536 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-15 02:55:44,536 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-15 02:55:44,536 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:44,536 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-15 02:55:44,536 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34681,1681527209133' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-15 02:55:44,537 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:44,537 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:44,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:44,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:55:44,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:55:44,537 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:44,537 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-15 02:55:44,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:55:44,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:55:44,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:55:44,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:44,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:55:44,538 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34681,1681527209133' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-15 02:55:44,538 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@a446e45[Count = 0] remaining members to acquire global barrier 2023-04-15 02:55:44,538 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-15 02:55:44,538 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:44,539 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:44,539 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:44,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:44,539 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-15 02:55:44,539 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-15 02:55:44,539 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:44,539 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-15 02:55:44,539 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34681,1681527209133' in zk 2023-04-15 02:55:44,539 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:44,539 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-15 02:55:44,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:44,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:55:44,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:55:44,540 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-15 02:55:44,540 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-15 02:55:44,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:55:44,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:55:44,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:55:44,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:44,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:55:44,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:55:44,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:44,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34681,1681527209133': 2023-04-15 02:55:44,541 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34681,1681527209133' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-15 02:55:44,541 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-15 02:55:44,541 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-15 02:55:44,541 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-15 02:55:44,541 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-15 02:55:44,541 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-15 02:55:44,542 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:44,542 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:44,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:44,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:55:44,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:55:44,542 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-15 02:55:44,542 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:44,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:44,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:55:44,542 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:44,542 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-15 02:55:44,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-15 02:55:44,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:55:44,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:44,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:55:44,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:55:44,543 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:44,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:44,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:55:44,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:55:44,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:44,544 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:44,545 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:55:44,545 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-15 02:55:44,545 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-15 02:55:44,545 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:55:44,545 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:44,545 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-15 02:55:44,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-15 02:55:44,545 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-15 02:55:44,545 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-15 02:55:44,545 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-15 02:55:44,545 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-15 02:55:44,545 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:44,545 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-15 02:55:44,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-15 02:55:44,545 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:44,545 DEBUG [Thread-2739] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-15 02:55:44,545 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:44,545 DEBUG [Thread-2739] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-15 02:55:44,645 DEBUG [Thread-2739] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-15 02:55:44,646 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-15 02:55:44,647 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-15 02:55:44,648 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-15 02:55:44,648 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-15 02:55:44,648 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-15 02:55:44,648 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-15 02:55:44,648 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-15 02:55:44,648 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-15 02:55:44,648 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:44,648 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:55:44,649 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-15 02:55:44,649 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:44,649 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-15 02:55:44,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-15 02:55:44,649 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:44,649 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-15 02:55:44,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:55:44,650 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:44,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-15 02:55:44,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:55:44,650 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-15 02:55:44,651 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-15 02:55:44,651 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-15 02:55:44,651 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-15 02:55:44,651 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-15 02:55:44,651 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-15 02:55:44,651 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-15 02:55:44,651 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-15 02:55:44,651 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681527331158.a29fa882a9e3f50c391e3aa08c5c5e72. 2023-04-15 02:55:44,651 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681527331158.a29fa882a9e3f50c391e3aa08c5c5e72. started... 2023-04-15 02:55:44,651 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681527331158.a29fa882a9e3f50c391e3aa08c5c5e72. as already flushing 2023-04-15 02:55:44,651 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681527331158.a29fa882a9e3f50c391e3aa08c5c5e72. 2023-04-15 02:55:44,651 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-15 02:55:44,651 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-15 02:55:44,651 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:44,651 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-15 02:55:44,651 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34681,1681527209133' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-15 02:55:44,652 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:44,652 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:44,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:44,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:55:44,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:55:44,652 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:44,652 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-15 02:55:44,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:55:44,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:55:44,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:55:44,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:44,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:55:44,653 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34681,1681527209133' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-15 02:55:44,653 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@52bf06a7[Count = 0] remaining members to acquire global barrier 2023-04-15 02:55:44,653 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-15 02:55:44,653 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:44,654 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:44,654 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:44,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:44,654 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-15 02:55:44,654 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-15 02:55:44,654 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34681,1681527209133' in zk 2023-04-15 02:55:44,654 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:44,654 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-15 02:55:44,655 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:44,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:44,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:55:44,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:55:44,655 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-15 02:55:44,655 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-15 02:55:44,655 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-15 02:55:44,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:55:44,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:55:44,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:55:44,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:44,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:55:44,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:55:44,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:44,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34681,1681527209133': 2023-04-15 02:55:44,657 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34681,1681527209133' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-15 02:55:44,657 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-15 02:55:44,657 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-15 02:55:44,657 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-15 02:55:44,657 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-15 02:55:44,657 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-15 02:55:44,658 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:44,658 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:44,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:44,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:55:44,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:55:44,658 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:44,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:44,658 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-15 02:55:44,659 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:44,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:55:44,659 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-15 02:55:44,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-15 02:55:44,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:55:44,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:44,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:55:44,659 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:44,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:55:44,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:44,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:55:44,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:55:44,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:44,660 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:44,660 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-15 02:55:44,660 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:55:44,661 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-15 02:55:44,661 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-15 02:55:44,661 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-15 02:55:44,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-15 02:55:44,661 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-15 02:55:44,661 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-15 02:55:44,661 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:55:44,661 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:44,661 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-15 02:55:44,661 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-15 02:55:44,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-15 02:55:44,661 DEBUG [Thread-2739] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-15 02:55:44,661 DEBUG [Thread-2739] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-15 02:55:44,661 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:44,661 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:44,661 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:44,663 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a29fa882a9e3f50c391e3aa08c5c5e72, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:55:44,663 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34681] ipc.CallRunner(144): callId: 141 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:40234 deadline: 1681527404663, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a29fa882a9e3f50c391e3aa08c5c5e72, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:44,669 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a29fa882a9e3f50c391e3aa08c5c5e72, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:55:44,669 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34681] ipc.CallRunner(144): callId: 153 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:40210 deadline: 1681527404666, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a29fa882a9e3f50c391e3aa08c5c5e72, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:44,675 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a29fa882a9e3f50c391e3aa08c5c5e72, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:55:44,675 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34681] ipc.CallRunner(144): callId: 153 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:40192 deadline: 1681527404670, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a29fa882a9e3f50c391e3aa08c5c5e72, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:44,684 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=271 (bloomFilter=true), to=hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/a29fa882a9e3f50c391e3aa08c5c5e72/.tmp/C/e0509719941d4e018f982030278d5c8c 2023-04-15 02:55:44,687 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/a29fa882a9e3f50c391e3aa08c5c5e72/.tmp/A/6637bb1f67c841e3a1430bf06afb4102 as hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/a29fa882a9e3f50c391e3aa08c5c5e72/A/6637bb1f67c841e3a1430bf06afb4102 2023-04-15 02:55:44,690 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/a29fa882a9e3f50c391e3aa08c5c5e72/A/6637bb1f67c841e3a1430bf06afb4102, entries=200, sequenceid=271, filesize=14.3 K 2023-04-15 02:55:44,690 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/a29fa882a9e3f50c391e3aa08c5c5e72/.tmp/B/6300d8f8bbcd462081165d7f3228ba43 as hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/a29fa882a9e3f50c391e3aa08c5c5e72/B/6300d8f8bbcd462081165d7f3228ba43 2023-04-15 02:55:44,693 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/a29fa882a9e3f50c391e3aa08c5c5e72/B/6300d8f8bbcd462081165d7f3228ba43, entries=150, sequenceid=271, filesize=11.9 K 2023-04-15 02:55:44,693 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/a29fa882a9e3f50c391e3aa08c5c5e72/.tmp/C/e0509719941d4e018f982030278d5c8c as hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/a29fa882a9e3f50c391e3aa08c5c5e72/C/e0509719941d4e018f982030278d5c8c 2023-04-15 02:55:44,696 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/a29fa882a9e3f50c391e3aa08c5c5e72/C/e0509719941d4e018f982030278d5c8c, entries=150, sequenceid=271, filesize=11.9 K 2023-04-15 02:55:44,696 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=134.18 KB/137400 for a29fa882a9e3f50c391e3aa08c5c5e72 in 1234ms, sequenceid=271, compaction requested=true 2023-04-15 02:55:44,696 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for a29fa882a9e3f50c391e3aa08c5c5e72: 2023-04-15 02:55:44,696 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-15 02:55:44,696 DEBUG [RS:0;jenkins-hbase20:34681-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-15 02:55:44,696 DEBUG [RS:0;jenkins-hbase20:34681-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-15 02:55:44,696 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-15 02:55:44,696 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-15 02:55:44,697 DEBUG [RS:0;jenkins-hbase20:34681-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 39226 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-15 02:55:44,697 DEBUG [RS:0;jenkins-hbase20:34681-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36786 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-15 02:55:44,697 DEBUG [RS:0;jenkins-hbase20:34681-shortCompactions-0] regionserver.HStore(1912): a29fa882a9e3f50c391e3aa08c5c5e72/A is initiating minor compaction (all files) 2023-04-15 02:55:44,697 DEBUG [RS:0;jenkins-hbase20:34681-longCompactions-0] regionserver.HStore(1912): a29fa882a9e3f50c391e3aa08c5c5e72/B is initiating minor compaction (all files) 2023-04-15 02:55:44,697 INFO [RS:0;jenkins-hbase20:34681-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of a29fa882a9e3f50c391e3aa08c5c5e72/A in TestAcidGuarantees,,1681527331158.a29fa882a9e3f50c391e3aa08c5c5e72. 2023-04-15 02:55:44,697 INFO [RS:0;jenkins-hbase20:34681-longCompactions-0] regionserver.HRegion(2259): Starting compaction of a29fa882a9e3f50c391e3aa08c5c5e72/B in TestAcidGuarantees,,1681527331158.a29fa882a9e3f50c391e3aa08c5c5e72. 2023-04-15 02:55:44,697 INFO [RS:0;jenkins-hbase20:34681-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/a29fa882a9e3f50c391e3aa08c5c5e72/A/bf1a5b878def417280c39297674887c5, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/a29fa882a9e3f50c391e3aa08c5c5e72/A/2791a854894341ac90b837693a2a8125, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/a29fa882a9e3f50c391e3aa08c5c5e72/A/6637bb1f67c841e3a1430bf06afb4102] into tmpdir=hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/a29fa882a9e3f50c391e3aa08c5c5e72/.tmp, totalSize=38.3 K 2023-04-15 02:55:44,697 INFO [RS:0;jenkins-hbase20:34681-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/a29fa882a9e3f50c391e3aa08c5c5e72/B/0173ef12b1354efb842e1ce4695fc4cd, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/a29fa882a9e3f50c391e3aa08c5c5e72/B/13cb677837cb482b924e6abbb42c6a41, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/a29fa882a9e3f50c391e3aa08c5c5e72/B/6300d8f8bbcd462081165d7f3228ba43] into tmpdir=hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/a29fa882a9e3f50c391e3aa08c5c5e72/.tmp, totalSize=35.9 K 2023-04-15 02:55:44,698 DEBUG [RS:0;jenkins-hbase20:34681-shortCompactions-0] compactions.Compactor(207): Compacting bf1a5b878def417280c39297674887c5, keycount=150, bloomtype=ROW, size=12.2 K, encoding=NONE, compression=NONE, seqNum=222, earliestPutTs=1681527339129 2023-04-15 02:55:44,698 DEBUG [RS:0;jenkins-hbase20:34681-longCompactions-0] compactions.Compactor(207): Compacting 0173ef12b1354efb842e1ce4695fc4cd, keycount=150, bloomtype=ROW, size=12.2 K, encoding=NONE, compression=NONE, seqNum=222, earliestPutTs=1681527339129 2023-04-15 02:55:44,698 DEBUG [RS:0;jenkins-hbase20:34681-shortCompactions-0] compactions.Compactor(207): Compacting 2791a854894341ac90b837693a2a8125, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=258, earliestPutTs=1681527341320 2023-04-15 02:55:44,698 DEBUG [RS:0;jenkins-hbase20:34681-longCompactions-0] compactions.Compactor(207): Compacting 13cb677837cb482b924e6abbb42c6a41, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=258, earliestPutTs=1681527341320 2023-04-15 02:55:44,698 DEBUG [RS:0;jenkins-hbase20:34681-shortCompactions-0] compactions.Compactor(207): Compacting 6637bb1f67c841e3a1430bf06afb4102, keycount=200, bloomtype=ROW, size=14.3 K, encoding=NONE, compression=NONE, seqNum=271, earliestPutTs=1681527343457 2023-04-15 02:55:44,698 DEBUG [RS:0;jenkins-hbase20:34681-longCompactions-0] compactions.Compactor(207): Compacting 6300d8f8bbcd462081165d7f3228ba43, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=271, earliestPutTs=1681527343457 2023-04-15 02:55:44,703 INFO [RS:0;jenkins-hbase20:34681-longCompactions-0] throttle.PressureAwareThroughputController(145): a29fa882a9e3f50c391e3aa08c5c5e72#B#compaction#395 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-15 02:55:44,703 INFO [RS:0;jenkins-hbase20:34681-shortCompactions-0] throttle.PressureAwareThroughputController(145): a29fa882a9e3f50c391e3aa08c5c5e72#A#compaction#396 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-15 02:55:44,761 DEBUG [Thread-2739] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-15 02:55:44,762 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-15 02:55:44,763 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-15 02:55:44,764 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-15 02:55:44,764 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-15 02:55:44,764 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-15 02:55:44,764 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-15 02:55:44,764 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-15 02:55:44,764 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-15 02:55:44,764 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:44,764 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:55:44,765 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:44,765 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-15 02:55:44,765 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-15 02:55:44,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-15 02:55:44,765 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:44,765 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-15 02:55:44,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:55:44,766 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:44,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-15 02:55:44,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:55:44,766 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-15 02:55:44,766 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-15 02:55:44,766 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-15 02:55:44,766 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-15 02:55:44,766 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-15 02:55:44,766 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-15 02:55:44,766 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-15 02:55:44,766 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-15 02:55:44,766 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681527331158.a29fa882a9e3f50c391e3aa08c5c5e72. 2023-04-15 02:55:44,766 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681527331158.a29fa882a9e3f50c391e3aa08c5c5e72. started... 2023-04-15 02:55:44,767 INFO [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing a29fa882a9e3f50c391e3aa08c5c5e72 3/3 column families, dataSize=134.18 KB heapSize=380.30 KB 2023-04-15 02:55:44,767 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a29fa882a9e3f50c391e3aa08c5c5e72, store=A 2023-04-15 02:55:44,767 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-15 02:55:44,767 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a29fa882a9e3f50c391e3aa08c5c5e72, store=B 2023-04-15 02:55:44,767 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-15 02:55:44,767 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a29fa882a9e3f50c391e3aa08c5c5e72, store=C 2023-04-15 02:55:44,767 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-15 02:55:45,115 DEBUG [RS:0;jenkins-hbase20:34681-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/a29fa882a9e3f50c391e3aa08c5c5e72/.tmp/A/cdcb1334de5b4aceb8a27f5d37d7cc43 as hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/a29fa882a9e3f50c391e3aa08c5c5e72/A/cdcb1334de5b4aceb8a27f5d37d7cc43 2023-04-15 02:55:45,115 DEBUG [RS:0;jenkins-hbase20:34681-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/a29fa882a9e3f50c391e3aa08c5c5e72/.tmp/B/7876501289af46cfbf6bfb8b5d2d89f2 as hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/a29fa882a9e3f50c391e3aa08c5c5e72/B/7876501289af46cfbf6bfb8b5d2d89f2 2023-04-15 02:55:45,118 INFO [RS:0;jenkins-hbase20:34681-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in a29fa882a9e3f50c391e3aa08c5c5e72/A of a29fa882a9e3f50c391e3aa08c5c5e72 into cdcb1334de5b4aceb8a27f5d37d7cc43(size=12.5 K), total size for store is 12.5 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-15 02:55:45,118 DEBUG [RS:0;jenkins-hbase20:34681-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for a29fa882a9e3f50c391e3aa08c5c5e72: 2023-04-15 02:55:45,118 INFO [RS:0;jenkins-hbase20:34681-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681527331158.a29fa882a9e3f50c391e3aa08c5c5e72., storeName=a29fa882a9e3f50c391e3aa08c5c5e72/A, priority=13, startTime=1681527344696; duration=0sec 2023-04-15 02:55:45,118 DEBUG [RS:0;jenkins-hbase20:34681-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-15 02:55:45,118 INFO [RS:0;jenkins-hbase20:34681-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in a29fa882a9e3f50c391e3aa08c5c5e72/B of a29fa882a9e3f50c391e3aa08c5c5e72 into 7876501289af46cfbf6bfb8b5d2d89f2(size=12.5 K), total size for store is 12.5 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-15 02:55:45,118 DEBUG [RS:0;jenkins-hbase20:34681-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-15 02:55:45,119 DEBUG [RS:0;jenkins-hbase20:34681-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for a29fa882a9e3f50c391e3aa08c5c5e72: 2023-04-15 02:55:45,119 INFO [RS:0;jenkins-hbase20:34681-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681527331158.a29fa882a9e3f50c391e3aa08c5c5e72., storeName=a29fa882a9e3f50c391e3aa08c5c5e72/B, priority=13, startTime=1681527344696; duration=0sec 2023-04-15 02:55:45,119 DEBUG [RS:0;jenkins-hbase20:34681-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-15 02:55:45,121 DEBUG [RS:0;jenkins-hbase20:34681-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36786 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-15 02:55:45,121 DEBUG [RS:0;jenkins-hbase20:34681-shortCompactions-0] regionserver.HStore(1912): a29fa882a9e3f50c391e3aa08c5c5e72/C is initiating minor compaction (all files) 2023-04-15 02:55:45,121 INFO [RS:0;jenkins-hbase20:34681-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of a29fa882a9e3f50c391e3aa08c5c5e72/C in TestAcidGuarantees,,1681527331158.a29fa882a9e3f50c391e3aa08c5c5e72. 2023-04-15 02:55:45,121 INFO [RS:0;jenkins-hbase20:34681-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/a29fa882a9e3f50c391e3aa08c5c5e72/C/258d4421a1fe497f89a4071b7edd4f2a, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/a29fa882a9e3f50c391e3aa08c5c5e72/C/f7bec9a155c544618f64cf938f34ced0, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/a29fa882a9e3f50c391e3aa08c5c5e72/C/e0509719941d4e018f982030278d5c8c] into tmpdir=hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/a29fa882a9e3f50c391e3aa08c5c5e72/.tmp, totalSize=35.9 K 2023-04-15 02:55:45,122 DEBUG [RS:0;jenkins-hbase20:34681-shortCompactions-0] compactions.Compactor(207): Compacting 258d4421a1fe497f89a4071b7edd4f2a, keycount=150, bloomtype=ROW, size=12.2 K, encoding=NONE, compression=NONE, seqNum=222, earliestPutTs=1681527339129 2023-04-15 02:55:45,122 DEBUG [RS:0;jenkins-hbase20:34681-shortCompactions-0] compactions.Compactor(207): Compacting f7bec9a155c544618f64cf938f34ced0, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=258, earliestPutTs=1681527341320 2023-04-15 02:55:45,122 DEBUG [RS:0;jenkins-hbase20:34681-shortCompactions-0] compactions.Compactor(207): Compacting e0509719941d4e018f982030278d5c8c, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=271, earliestPutTs=1681527343457 2023-04-15 02:55:45,126 INFO [RS:0;jenkins-hbase20:34681-shortCompactions-0] throttle.PressureAwareThroughputController(145): a29fa882a9e3f50c391e3aa08c5c5e72#C#compaction#398 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-15 02:55:45,173 INFO [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=307 (bloomFilter=true), to=hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/a29fa882a9e3f50c391e3aa08c5c5e72/.tmp/A/1e6fdf63e00641cf974a6d51d0c9c42f 2023-04-15 02:55:45,181 INFO [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=307 (bloomFilter=true), to=hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/a29fa882a9e3f50c391e3aa08c5c5e72/.tmp/B/47c3b4716c284989b651f97044b9111d 2023-04-15 02:55:45,192 INFO [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=307 (bloomFilter=true), to=hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/a29fa882a9e3f50c391e3aa08c5c5e72/.tmp/C/8e9b2ace06874a618b994b9577a16160 2023-04-15 02:55:45,195 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/a29fa882a9e3f50c391e3aa08c5c5e72/.tmp/A/1e6fdf63e00641cf974a6d51d0c9c42f as hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/a29fa882a9e3f50c391e3aa08c5c5e72/A/1e6fdf63e00641cf974a6d51d0c9c42f 2023-04-15 02:55:45,198 INFO [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/a29fa882a9e3f50c391e3aa08c5c5e72/A/1e6fdf63e00641cf974a6d51d0c9c42f, entries=150, sequenceid=307, filesize=11.9 K 2023-04-15 02:55:45,199 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/a29fa882a9e3f50c391e3aa08c5c5e72/.tmp/B/47c3b4716c284989b651f97044b9111d as hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/a29fa882a9e3f50c391e3aa08c5c5e72/B/47c3b4716c284989b651f97044b9111d 2023-04-15 02:55:45,201 INFO [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/a29fa882a9e3f50c391e3aa08c5c5e72/B/47c3b4716c284989b651f97044b9111d, entries=150, sequenceid=307, filesize=11.9 K 2023-04-15 02:55:45,202 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/a29fa882a9e3f50c391e3aa08c5c5e72/.tmp/C/8e9b2ace06874a618b994b9577a16160 as hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/a29fa882a9e3f50c391e3aa08c5c5e72/C/8e9b2ace06874a618b994b9577a16160 2023-04-15 02:55:45,205 INFO [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/a29fa882a9e3f50c391e3aa08c5c5e72/C/8e9b2ace06874a618b994b9577a16160, entries=150, sequenceid=307, filesize=11.9 K 2023-04-15 02:55:45,206 INFO [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~134.18 KB/137400, heapSize ~380.25 KB/389376, currentSize=0 B/0 for a29fa882a9e3f50c391e3aa08c5c5e72 in 440ms, sequenceid=307, compaction requested=false 2023-04-15 02:55:45,206 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for a29fa882a9e3f50c391e3aa08c5c5e72: 2023-04-15 02:55:45,206 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681527331158.a29fa882a9e3f50c391e3aa08c5c5e72. 2023-04-15 02:55:45,206 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-15 02:55:45,206 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-15 02:55:45,206 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:45,210 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-15 02:55:45,210 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34681,1681527209133' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-15 02:55:45,211 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:45,211 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:45,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:45,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:55:45,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:55:45,212 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:45,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:55:45,212 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-15 02:55:45,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:55:45,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:55:45,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:45,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:55:45,213 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34681,1681527209133' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-15 02:55:45,213 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4cea28cf[Count = 0] remaining members to acquire global barrier 2023-04-15 02:55:45,213 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-15 02:55:45,213 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:45,214 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:45,214 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:45,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:45,214 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-15 02:55:45,214 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-15 02:55:45,214 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34681,1681527209133' in zk 2023-04-15 02:55:45,214 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:45,214 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-15 02:55:45,215 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-15 02:55:45,215 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:45,215 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-15 02:55:45,215 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-15 02:55:45,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:45,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:55:45,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:55:45,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:55:45,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:55:45,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:55:45,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:45,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:55:45,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:55:45,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:45,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34681,1681527209133': 2023-04-15 02:55:45,220 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34681,1681527209133' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-15 02:55:45,220 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-15 02:55:45,220 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-15 02:55:45,220 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-15 02:55:45,220 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-15 02:55:45,220 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-15 02:55:45,222 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:45,222 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:45,222 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-15 02:55:45,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:45,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:55:45,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:55:45,222 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:45,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:45,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:55:45,223 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-15 02:55:45,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-15 02:55:45,223 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:45,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:45,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:55:45,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:55:45,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:55:45,225 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:45,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:45,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:55:45,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:55:45,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:45,230 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:45,230 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-15 02:55:45,231 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-15 02:55:45,231 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-15 02:55:45,231 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-15 02:55:45,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-15 02:55:45,231 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:55:45,231 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:55:45,231 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-15 02:55:45,231 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-15 02:55:45,231 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-15 02:55:45,231 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:45,231 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-15 02:55:45,231 DEBUG [Thread-2739] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-15 02:55:45,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-15 02:55:45,231 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:45,231 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:45,231 DEBUG [Thread-2739] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-15 02:55:45,231 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:45,331 DEBUG [Thread-2739] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-15 02:55:45,332 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-15 02:55:45,333 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-15 02:55:45,334 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-15 02:55:45,334 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-15 02:55:45,334 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-15 02:55:45,334 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-15 02:55:45,334 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-15 02:55:45,334 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-15 02:55:45,334 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:45,334 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:55:45,335 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-15 02:55:45,335 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-15 02:55:45,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-15 02:55:45,335 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:45,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:55:45,335 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:45,335 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-15 02:55:45,335 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:45,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-15 02:55:45,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:55:45,336 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-15 02:55:45,336 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-15 02:55:45,336 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-15 02:55:45,336 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-15 02:55:45,336 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-15 02:55:45,336 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-15 02:55:45,336 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-15 02:55:45,336 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-15 02:55:45,336 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681527331158.a29fa882a9e3f50c391e3aa08c5c5e72. 2023-04-15 02:55:45,336 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681527331158.a29fa882a9e3f50c391e3aa08c5c5e72. started... 2023-04-15 02:55:45,336 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for a29fa882a9e3f50c391e3aa08c5c5e72: 2023-04-15 02:55:45,337 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681527331158.a29fa882a9e3f50c391e3aa08c5c5e72. 2023-04-15 02:55:45,337 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-15 02:55:45,337 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-15 02:55:45,337 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:45,337 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-15 02:55:45,337 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34681,1681527209133' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-15 02:55:45,337 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:45,337 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:45,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:45,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:55:45,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:55:45,338 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:45,338 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-15 02:55:45,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:55:45,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:55:45,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:55:45,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:45,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:55:45,339 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34681,1681527209133' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-15 02:55:45,339 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-15 02:55:45,339 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7b48a985[Count = 0] remaining members to acquire global barrier 2023-04-15 02:55:45,339 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:45,340 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:45,340 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:45,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:45,340 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-15 02:55:45,340 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:45,340 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-15 02:55:45,340 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-15 02:55:45,340 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34681,1681527209133' in zk 2023-04-15 02:55:45,341 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:45,341 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-15 02:55:45,341 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-15 02:55:45,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:45,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:55:45,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:55:45,341 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-15 02:55:45,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:55:45,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:55:45,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:55:45,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:45,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:55:45,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:55:45,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:45,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34681,1681527209133': 2023-04-15 02:55:45,343 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34681,1681527209133' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-15 02:55:45,343 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-15 02:55:45,343 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-15 02:55:45,343 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-15 02:55:45,343 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-15 02:55:45,343 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-15 02:55:45,343 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:45,344 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:45,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:45,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:55:45,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:55:45,344 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:45,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:45,344 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-15 02:55:45,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:55:45,344 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:45,344 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-15 02:55:45,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-15 02:55:45,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:55:45,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:45,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:55:45,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:55:45,345 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:45,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:45,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:55:45,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:55:45,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:45,346 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:45,346 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-15 02:55:45,346 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:55:45,346 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-15 02:55:45,346 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-15 02:55:45,346 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:55:45,346 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-15 02:55:45,347 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-15 02:55:45,346 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-15 02:55:45,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-15 02:55:45,346 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-15 02:55:45,347 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:45,347 DEBUG [Thread-2739] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-15 02:55:45,347 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:45,347 DEBUG [Thread-2739] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-15 02:55:45,347 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:45,347 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:45,347 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-15 02:55:45,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-15 02:55:45,447 DEBUG [Thread-2739] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-15 02:55:45,447 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-15 02:55:45,449 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-15 02:55:45,449 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-15 02:55:45,450 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-15 02:55:45,450 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-15 02:55:45,450 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-15 02:55:45,450 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-15 02:55:45,450 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-15 02:55:45,450 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:45,450 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:55:45,458 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-15 02:55:45,458 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:45,458 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-15 02:55:45,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-15 02:55:45,459 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:45,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:55:45,459 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-15 02:55:45,459 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:45,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-15 02:55:45,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:55:45,459 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-15 02:55:45,459 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-15 02:55:45,460 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-15 02:55:45,460 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-15 02:55:45,460 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-15 02:55:45,460 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-15 02:55:45,460 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-15 02:55:45,460 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-15 02:55:45,460 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681527331158.a29fa882a9e3f50c391e3aa08c5c5e72. 2023-04-15 02:55:45,460 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681527331158.a29fa882a9e3f50c391e3aa08c5c5e72. started... 2023-04-15 02:55:45,460 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for a29fa882a9e3f50c391e3aa08c5c5e72: 2023-04-15 02:55:45,460 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681527331158.a29fa882a9e3f50c391e3aa08c5c5e72. 2023-04-15 02:55:45,460 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-15 02:55:45,460 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-15 02:55:45,460 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:45,460 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-15 02:55:45,460 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34681,1681527209133' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-15 02:55:45,464 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:45,464 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:45,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:45,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:55:45,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:55:45,464 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:45,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:55:45,464 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-15 02:55:45,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:55:45,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:55:45,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:45,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:55:45,465 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34681,1681527209133' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-15 02:55:45,465 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5257c6ef[Count = 0] remaining members to acquire global barrier 2023-04-15 02:55:45,465 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-15 02:55:45,465 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:45,466 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:45,466 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:45,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:45,466 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-15 02:55:45,466 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-15 02:55:45,466 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34681,1681527209133' in zk 2023-04-15 02:55:45,466 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:45,466 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-15 02:55:45,467 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-15 02:55:45,467 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:45,467 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-15 02:55:45,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:45,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:55:45,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:55:45,467 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-15 02:55:45,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:55:45,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:55:45,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:55:45,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:45,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:55:45,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:55:45,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:45,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34681,1681527209133': 2023-04-15 02:55:45,469 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34681,1681527209133' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-15 02:55:45,469 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-15 02:55:45,469 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-15 02:55:45,469 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-15 02:55:45,469 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-15 02:55:45,469 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-15 02:55:45,470 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:45,470 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:45,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:45,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:55:45,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:55:45,470 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:45,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:45,470 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-15 02:55:45,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:55:45,470 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-15 02:55:45,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-15 02:55:45,471 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:45,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:55:45,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:45,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:55:45,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:55:45,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:45,472 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:45,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:55:45,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:55:45,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:45,474 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:45,474 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:55:45,474 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:55:45,474 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-15 02:55:45,474 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-15 02:55:45,474 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-15 02:55:45,474 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-15 02:55:45,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-15 02:55:45,474 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:45,474 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-15 02:55:45,474 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-15 02:55:45,474 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:45,474 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-15 02:55:45,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-15 02:55:45,474 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:45,474 DEBUG [Thread-2739] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-15 02:55:45,475 DEBUG [Thread-2739] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-15 02:55:45,475 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:45,534 DEBUG [RS:0;jenkins-hbase20:34681-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/a29fa882a9e3f50c391e3aa08c5c5e72/.tmp/C/2a2de79b0f804d0aa7c763bfa19f47b1 as hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/a29fa882a9e3f50c391e3aa08c5c5e72/C/2a2de79b0f804d0aa7c763bfa19f47b1 2023-04-15 02:55:45,537 INFO [RS:0;jenkins-hbase20:34681-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in a29fa882a9e3f50c391e3aa08c5c5e72/C of a29fa882a9e3f50c391e3aa08c5c5e72 into 2a2de79b0f804d0aa7c763bfa19f47b1(size=12.5 K), total size for store is 24.4 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-15 02:55:45,537 DEBUG [RS:0;jenkins-hbase20:34681-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for a29fa882a9e3f50c391e3aa08c5c5e72: 2023-04-15 02:55:45,538 INFO [RS:0;jenkins-hbase20:34681-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681527331158.a29fa882a9e3f50c391e3aa08c5c5e72., storeName=a29fa882a9e3f50c391e3aa08c5c5e72/C, priority=13, startTime=1681527344696; duration=0sec 2023-04-15 02:55:45,538 DEBUG [RS:0;jenkins-hbase20:34681-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-15 02:55:45,575 DEBUG [Thread-2739] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-15 02:55:45,575 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-15 02:55:45,576 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-15 02:55:45,577 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-15 02:55:45,577 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-15 02:55:45,577 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-15 02:55:45,577 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-15 02:55:45,577 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-15 02:55:45,577 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-15 02:55:45,578 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:45,578 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:55:45,578 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-15 02:55:45,578 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:45,578 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-15 02:55:45,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-15 02:55:45,579 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:45,579 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-15 02:55:45,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:55:45,579 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:45,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-15 02:55:45,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:55:45,579 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-15 02:55:45,580 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-15 02:55:45,580 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-15 02:55:45,580 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-15 02:55:45,580 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-15 02:55:45,580 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-15 02:55:45,580 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-15 02:55:45,580 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-15 02:55:45,580 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681527331158.a29fa882a9e3f50c391e3aa08c5c5e72. 2023-04-15 02:55:45,580 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681527331158.a29fa882a9e3f50c391e3aa08c5c5e72. started... 2023-04-15 02:55:45,580 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for a29fa882a9e3f50c391e3aa08c5c5e72: 2023-04-15 02:55:45,580 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681527331158.a29fa882a9e3f50c391e3aa08c5c5e72. 2023-04-15 02:55:45,580 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-15 02:55:45,580 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-15 02:55:45,580 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:45,581 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-15 02:55:45,581 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34681,1681527209133' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-15 02:55:45,581 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:45,581 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:45,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:45,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:55:45,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:55:45,582 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:45,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:55:45,582 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-15 02:55:45,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:55:45,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:55:45,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:45,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:55:45,583 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34681,1681527209133' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-15 02:55:45,583 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@78f7683c[Count = 0] remaining members to acquire global barrier 2023-04-15 02:55:45,583 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-15 02:55:45,583 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:45,584 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:45,584 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:45,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:45,584 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-15 02:55:45,584 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-15 02:55:45,584 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:45,584 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-15 02:55:45,584 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34681,1681527209133' in zk 2023-04-15 02:55:45,585 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:45,585 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-15 02:55:45,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:45,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:55:45,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:55:45,585 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-15 02:55:45,585 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-15 02:55:45,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:55:45,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:55:45,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:55:45,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:45,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:55:45,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:55:45,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:45,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34681,1681527209133': 2023-04-15 02:55:45,587 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34681,1681527209133' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-15 02:55:45,587 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-15 02:55:45,587 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-15 02:55:45,587 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-15 02:55:45,587 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-15 02:55:45,587 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-15 02:55:45,588 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:45,588 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-15 02:55:45,588 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:45,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:45,588 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:45,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:45,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:55:45,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:55:45,588 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-15 02:55:45,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-15 02:55:45,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:55:45,589 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:45,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:45,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:55:45,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:55:45,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:55:45,590 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:45,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:45,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:55:45,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:55:45,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:45,591 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-15 02:55:45,591 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:45,592 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:55:45,592 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:55:45,592 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:45,592 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-15 02:55:45,592 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-15 02:55:45,592 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-15 02:55:45,592 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-15 02:55:45,592 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-15 02:55:45,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-15 02:55:45,592 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:45,592 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-15 02:55:45,592 DEBUG [Thread-2739] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-15 02:55:45,592 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:45,592 DEBUG [Thread-2739] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-15 02:55:45,592 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:45,592 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-15 02:55:45,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-15 02:55:45,677 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34681] regionserver.HRegion(9158): Flush requested on a29fa882a9e3f50c391e3aa08c5c5e72 2023-04-15 02:55:45,677 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing a29fa882a9e3f50c391e3aa08c5c5e72 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-15 02:55:45,677 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a29fa882a9e3f50c391e3aa08c5c5e72, store=A 2023-04-15 02:55:45,677 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-15 02:55:45,678 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a29fa882a9e3f50c391e3aa08c5c5e72, store=B 2023-04-15 02:55:45,678 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-15 02:55:45,678 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a29fa882a9e3f50c391e3aa08c5c5e72, store=C 2023-04-15 02:55:45,678 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-15 02:55:45,685 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=323 (bloomFilter=true), to=hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/a29fa882a9e3f50c391e3aa08c5c5e72/.tmp/A/cb1cebeccfd7464b84d6eb0a083c50c0 2023-04-15 02:55:45,692 DEBUG [Thread-2739] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-15 02:55:45,693 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-15 02:55:45,694 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-15 02:55:45,695 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-15 02:55:45,695 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-15 02:55:45,695 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-15 02:55:45,695 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-15 02:55:45,695 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-15 02:55:45,695 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-15 02:55:45,696 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:45,696 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:55:45,696 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-15 02:55:45,696 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:45,696 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-15 02:55:45,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-15 02:55:45,696 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:45,696 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-15 02:55:45,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:55:45,697 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:45,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-15 02:55:45,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:55:45,697 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-15 02:55:45,697 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-15 02:55:45,697 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-15 02:55:45,697 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-15 02:55:45,697 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-15 02:55:45,697 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-15 02:55:45,697 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-15 02:55:45,697 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681527331158.a29fa882a9e3f50c391e3aa08c5c5e72. 2023-04-15 02:55:45,697 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681527331158.a29fa882a9e3f50c391e3aa08c5c5e72. started... 2023-04-15 02:55:45,697 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-15 02:55:45,697 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681527331158.a29fa882a9e3f50c391e3aa08c5c5e72. as already flushing 2023-04-15 02:55:45,698 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681527331158.a29fa882a9e3f50c391e3aa08c5c5e72. 2023-04-15 02:55:45,698 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-15 02:55:45,698 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-15 02:55:45,698 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:45,698 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-15 02:55:45,698 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34681,1681527209133' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-15 02:55:45,698 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:45,698 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:45,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:45,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:55:45,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:55:45,699 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:45,699 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-15 02:55:45,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:55:45,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:55:45,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:55:45,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:45,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:55:45,700 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34681,1681527209133' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-15 02:55:45,700 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-15 02:55:45,700 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@65c68bbd[Count = 0] remaining members to acquire global barrier 2023-04-15 02:55:45,700 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:45,700 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:45,700 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:45,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:45,701 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-15 02:55:45,701 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-15 02:55:45,701 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34681,1681527209133' in zk 2023-04-15 02:55:45,701 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:45,701 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-15 02:55:45,701 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:45,701 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-15 02:55:45,701 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-15 02:55:45,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:45,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:55:45,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:55:45,701 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-15 02:55:45,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:55:45,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:55:45,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:55:45,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:45,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:55:45,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:55:45,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:45,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34681,1681527209133': 2023-04-15 02:55:45,703 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34681,1681527209133' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-15 02:55:45,703 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-15 02:55:45,703 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-15 02:55:45,703 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-15 02:55:45,703 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-15 02:55:45,703 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-15 02:55:45,704 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:45,704 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:45,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:45,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:55:45,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:55:45,704 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:45,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:45,704 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-15 02:55:45,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:55:45,704 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-15 02:55:45,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-15 02:55:45,704 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:45,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:45,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:55:45,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:55:45,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:55:45,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:45,706 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:45,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:55:45,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:55:45,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:45,707 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1681527345692, max timestamp=1681527345706Num uniques 859; 2023-04-15 02:55:45,708 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-8] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-15 02:55:45,708 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:45,708 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:55:45,708 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:55:45,708 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1681527345692, max timestamp=1681527345706Num uniques 859; 2023-04-15 02:55:45,708 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-6] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-15 02:55:45,708 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:45,708 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-15 02:55:45,708 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-15 02:55:45,708 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-15 02:55:45,708 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1681527345692, max timestamp=1681527345706Num uniques 859; 2023-04-15 02:55:45,708 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-2] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-15 02:55:45,708 DEBUG [Thread-2739] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-15 02:55:45,708 DEBUG [Thread-2739] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-15 02:55:45,708 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:45,708 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:45,708 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-15 02:55:45,708 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:45,708 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-15 02:55:45,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-15 02:55:45,708 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-15 02:55:45,709 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-15 02:55:45,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-15 02:55:45,749 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a29fa882a9e3f50c391e3aa08c5c5e72, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:55:45,749 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34681] ipc.CallRunner(144): callId: 162 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:40234 deadline: 1681527405741, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a29fa882a9e3f50c391e3aa08c5c5e72, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:45,749 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a29fa882a9e3f50c391e3aa08c5c5e72, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:55:45,749 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34681] ipc.CallRunner(144): callId: 166 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:40210 deadline: 1681527405747, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a29fa882a9e3f50c391e3aa08c5c5e72, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:45,753 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a29fa882a9e3f50c391e3aa08c5c5e72, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:55:45,754 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34681] ipc.CallRunner(144): callId: 165 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:40192 deadline: 1681527405748, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a29fa882a9e3f50c391e3aa08c5c5e72, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:45,808 DEBUG [Thread-2739] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-15 02:55:45,809 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-15 02:55:45,810 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-15 02:55:45,811 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-15 02:55:45,811 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-15 02:55:45,811 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-15 02:55:45,811 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-15 02:55:45,811 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-15 02:55:45,811 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-15 02:55:45,812 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:45,812 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:55:45,812 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:45,812 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-15 02:55:45,812 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-15 02:55:45,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-15 02:55:45,813 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:45,813 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-15 02:55:45,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:55:45,813 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:45,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-15 02:55:45,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:55:45,813 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-15 02:55:45,813 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-15 02:55:45,813 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-15 02:55:45,813 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-15 02:55:45,813 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-15 02:55:45,813 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-15 02:55:45,813 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-15 02:55:45,814 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-15 02:55:45,814 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681527331158.a29fa882a9e3f50c391e3aa08c5c5e72. 2023-04-15 02:55:45,814 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681527331158.a29fa882a9e3f50c391e3aa08c5c5e72. started... 2023-04-15 02:55:45,814 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681527331158.a29fa882a9e3f50c391e3aa08c5c5e72. as already flushing 2023-04-15 02:55:45,814 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681527331158.a29fa882a9e3f50c391e3aa08c5c5e72. 2023-04-15 02:55:45,814 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-15 02:55:45,814 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-15 02:55:45,814 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:45,814 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-15 02:55:45,814 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34681,1681527209133' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-15 02:55:45,815 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:45,815 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:45,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:45,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:55:45,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:55:45,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:55:45,815 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:45,815 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-15 02:55:45,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:55:45,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:55:45,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:45,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:55:45,816 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34681,1681527209133' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-15 02:55:45,816 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-15 02:55:45,816 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5b7c0201[Count = 0] remaining members to acquire global barrier 2023-04-15 02:55:45,816 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:45,817 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:45,817 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:45,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:45,817 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-15 02:55:45,817 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-15 02:55:45,817 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34681,1681527209133' in zk 2023-04-15 02:55:45,817 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:45,817 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-15 02:55:45,817 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:45,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:45,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:55:45,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:55:45,818 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-15 02:55:45,818 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-15 02:55:45,818 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-15 02:55:45,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:55:45,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:55:45,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:55:45,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:45,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:55:45,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:55:45,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:45,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34681,1681527209133': 2023-04-15 02:55:45,819 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34681,1681527209133' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-15 02:55:45,819 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-15 02:55:45,819 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-15 02:55:45,819 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-15 02:55:45,819 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-15 02:55:45,819 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-15 02:55:45,820 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:45,820 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:45,820 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-15 02:55:45,820 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:45,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:45,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:45,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:55:45,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:55:45,820 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-15 02:55:45,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-15 02:55:45,821 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:45,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:55:45,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:45,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:55:45,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:55:45,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:55:45,821 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:45,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:45,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:55:45,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:55:45,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:45,823 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:45,823 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-15 02:55:45,823 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:55:45,824 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-15 02:55:45,824 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-15 02:55:45,824 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:55:45,824 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:45,824 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:45,824 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:45,824 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-15 02:55:45,824 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-15 02:55:45,824 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-15 02:55:45,824 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-15 02:55:45,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-15 02:55:45,824 DEBUG [Thread-2739] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-15 02:55:45,824 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:45,824 DEBUG [Thread-2739] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-15 02:55:45,824 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-15 02:55:45,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-15 02:55:45,853 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a29fa882a9e3f50c391e3aa08c5c5e72, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:55:45,853 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34681] ipc.CallRunner(144): callId: 164 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:40234 deadline: 1681527405850, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a29fa882a9e3f50c391e3aa08c5c5e72, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:45,853 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a29fa882a9e3f50c391e3aa08c5c5e72, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:55:45,853 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34681] ipc.CallRunner(144): callId: 168 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:40210 deadline: 1681527405850, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a29fa882a9e3f50c391e3aa08c5c5e72, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:45,859 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a29fa882a9e3f50c391e3aa08c5c5e72, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:55:45,859 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34681] ipc.CallRunner(144): callId: 167 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:40192 deadline: 1681527405854, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a29fa882a9e3f50c391e3aa08c5c5e72, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:45,924 DEBUG [Thread-2739] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-15 02:55:45,925 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-15 02:55:45,926 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-15 02:55:45,927 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-15 02:55:45,927 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-15 02:55:45,927 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-15 02:55:45,927 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-15 02:55:45,927 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-15 02:55:45,927 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-15 02:55:45,927 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:45,927 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:55:45,928 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-15 02:55:45,928 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:45,928 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-15 02:55:45,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-15 02:55:45,928 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:45,928 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-15 02:55:45,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:55:45,929 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:45,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-15 02:55:45,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:55:45,929 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-15 02:55:45,929 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-15 02:55:45,929 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-15 02:55:45,929 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-15 02:55:45,929 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-15 02:55:45,929 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-15 02:55:45,929 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-15 02:55:45,929 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681527331158.a29fa882a9e3f50c391e3aa08c5c5e72. 2023-04-15 02:55:45,929 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681527331158.a29fa882a9e3f50c391e3aa08c5c5e72. started... 2023-04-15 02:55:45,930 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681527331158.a29fa882a9e3f50c391e3aa08c5c5e72. as already flushing 2023-04-15 02:55:45,930 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681527331158.a29fa882a9e3f50c391e3aa08c5c5e72. 2023-04-15 02:55:45,930 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-15 02:55:45,930 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-15 02:55:45,930 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:45,930 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-15 02:55:45,930 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34681,1681527209133' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-15 02:55:45,930 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:45,931 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:45,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:45,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:55:45,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:55:45,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:55:45,932 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:45,932 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-15 02:55:45,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:55:45,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:55:45,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:45,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:55:45,934 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34681,1681527209133' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-15 02:55:45,934 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-15 02:55:45,934 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3770c73f[Count = 0] remaining members to acquire global barrier 2023-04-15 02:55:45,934 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:45,935 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:45,935 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:45,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:45,935 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-15 02:55:45,935 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-15 02:55:45,935 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:45,935 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-15 02:55:45,935 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34681,1681527209133' in zk 2023-04-15 02:55:45,935 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-15 02:55:45,936 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:45,936 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-15 02:55:45,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:45,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:55:45,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:55:45,936 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-15 02:55:45,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:55:45,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:55:45,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:55:45,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:45,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:55:45,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:55:45,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:45,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34681,1681527209133': 2023-04-15 02:55:45,937 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34681,1681527209133' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-15 02:55:45,937 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-15 02:55:45,937 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-15 02:55:45,937 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-15 02:55:45,937 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-15 02:55:45,937 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-15 02:55:45,938 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:45,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:45,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:55:45,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:55:45,939 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:45,939 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-15 02:55:45,939 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:45,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:45,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:55:45,939 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-15 02:55:45,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-15 02:55:45,939 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:45,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:55:45,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:45,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:55:45,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:55:45,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:45,940 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:45,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:55:45,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:55:45,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:45,941 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:45,941 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:55:45,941 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-15 02:55:45,941 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:55:45,941 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-15 02:55:45,941 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-15 02:55:45,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-15 02:55:45,941 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-15 02:55:45,941 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:45,941 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:45,941 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:45,941 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:45,941 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-15 02:55:45,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-15 02:55:45,942 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-15 02:55:45,942 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-15 02:55:45,942 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-15 02:55:45,942 DEBUG [Thread-2739] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-15 02:55:45,942 DEBUG [Thread-2739] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-15 02:55:46,042 DEBUG [Thread-2739] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-15 02:55:46,042 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-15 02:55:46,044 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-15 02:55:46,044 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-15 02:55:46,045 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-15 02:55:46,045 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-15 02:55:46,045 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-15 02:55:46,045 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-15 02:55:46,045 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-15 02:55:46,045 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:46,045 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:55:46,046 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:46,046 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-15 02:55:46,046 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-15 02:55:46,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-15 02:55:46,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:55:46,046 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:46,046 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-15 02:55:46,046 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:46,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-15 02:55:46,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:55:46,046 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-15 02:55:46,047 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-15 02:55:46,047 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-15 02:55:46,047 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-15 02:55:46,047 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-15 02:55:46,047 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-15 02:55:46,047 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-15 02:55:46,047 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-15 02:55:46,047 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681527331158.a29fa882a9e3f50c391e3aa08c5c5e72. 2023-04-15 02:55:46,047 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681527331158.a29fa882a9e3f50c391e3aa08c5c5e72. started... 2023-04-15 02:55:46,047 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681527331158.a29fa882a9e3f50c391e3aa08c5c5e72. as already flushing 2023-04-15 02:55:46,047 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681527331158.a29fa882a9e3f50c391e3aa08c5c5e72. 2023-04-15 02:55:46,047 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-15 02:55:46,047 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-15 02:55:46,047 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:46,047 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-15 02:55:46,047 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34681,1681527209133' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-15 02:55:46,048 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:46,048 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:46,048 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-15 02:55:46,049 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:46,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:46,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:55:46,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:55:46,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:55:46,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:55:46,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:55:46,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:46,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:55:46,050 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34681,1681527209133' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-15 02:55:46,050 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-15 02:55:46,050 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7485ef65[Count = 0] remaining members to acquire global barrier 2023-04-15 02:55:46,050 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:46,051 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:46,051 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:46,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:46,051 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-15 02:55:46,051 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-15 02:55:46,051 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:46,051 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-15 02:55:46,051 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34681,1681527209133' in zk 2023-04-15 02:55:46,052 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-15 02:55:46,052 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:46,052 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-15 02:55:46,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:46,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:55:46,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:55:46,052 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-15 02:55:46,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:55:46,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:55:46,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:55:46,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:46,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:55:46,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:55:46,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:46,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34681,1681527209133': 2023-04-15 02:55:46,054 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34681,1681527209133' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-15 02:55:46,054 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-15 02:55:46,054 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-15 02:55:46,054 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-15 02:55:46,054 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-15 02:55:46,054 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-15 02:55:46,054 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:46,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:46,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:55:46,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:55:46,055 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:46,055 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:46,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:46,055 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-15 02:55:46,055 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-15 02:55:46,055 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:46,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:55:46,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-15 02:55:46,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:55:46,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:46,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:55:46,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:55:46,057 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:46,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:46,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:55:46,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:55:46,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:46,059 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:46,059 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:55:46,059 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:55:46,059 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:46,059 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-15 02:55:46,059 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-15 02:55:46,059 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-15 02:55:46,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-15 02:55:46,059 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-15 02:55:46,059 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-15 02:55:46,060 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-15 02:55:46,059 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:46,060 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-15 02:55:46,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-15 02:55:46,060 DEBUG [Thread-2739] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-15 02:55:46,060 DEBUG [Thread-2739] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-15 02:55:46,059 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-15 02:55:46,060 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:46,060 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:46,060 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a29fa882a9e3f50c391e3aa08c5c5e72, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:55:46,060 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34681] ipc.CallRunner(144): callId: 170 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:40210 deadline: 1681527406054, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a29fa882a9e3f50c391e3aa08c5c5e72, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:46,060 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a29fa882a9e3f50c391e3aa08c5c5e72, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:55:46,061 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34681] ipc.CallRunner(144): callId: 166 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:40234 deadline: 1681527406055, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a29fa882a9e3f50c391e3aa08c5c5e72, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:46,065 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a29fa882a9e3f50c391e3aa08c5c5e72, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:55:46,065 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34681] ipc.CallRunner(144): callId: 169 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:40192 deadline: 1681527406060, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a29fa882a9e3f50c391e3aa08c5c5e72, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:46,093 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=323 (bloomFilter=true), to=hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/a29fa882a9e3f50c391e3aa08c5c5e72/.tmp/B/1f7949a7a669426ab732be2e19ad4f6d 2023-04-15 02:55:46,160 DEBUG [Thread-2739] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-15 02:55:46,160 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-15 02:55:46,162 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-15 02:55:46,162 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-15 02:55:46,162 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-15 02:55:46,163 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-15 02:55:46,163 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-15 02:55:46,163 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-15 02:55:46,163 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-15 02:55:46,163 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:46,163 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:55:46,163 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:46,163 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-15 02:55:46,164 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-15 02:55:46,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-15 02:55:46,164 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:46,164 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-15 02:55:46,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:55:46,165 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:46,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-15 02:55:46,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:55:46,166 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-15 02:55:46,166 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-15 02:55:46,166 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-15 02:55:46,166 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-15 02:55:46,166 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-15 02:55:46,166 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-15 02:55:46,166 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-15 02:55:46,166 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-15 02:55:46,166 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681527331158.a29fa882a9e3f50c391e3aa08c5c5e72. 2023-04-15 02:55:46,166 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681527331158.a29fa882a9e3f50c391e3aa08c5c5e72. started... 2023-04-15 02:55:46,166 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681527331158.a29fa882a9e3f50c391e3aa08c5c5e72. as already flushing 2023-04-15 02:55:46,166 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681527331158.a29fa882a9e3f50c391e3aa08c5c5e72. 2023-04-15 02:55:46,166 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-15 02:55:46,166 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-15 02:55:46,166 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:46,166 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-15 02:55:46,166 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34681,1681527209133' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-15 02:55:46,167 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:46,167 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:46,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:46,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:55:46,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:55:46,168 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:46,168 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-15 02:55:46,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:55:46,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:55:46,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:55:46,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:46,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:55:46,169 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34681,1681527209133' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-15 02:55:46,169 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-15 02:55:46,169 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@254f092b[Count = 0] remaining members to acquire global barrier 2023-04-15 02:55:46,169 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:46,169 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:46,169 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:46,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:46,169 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-15 02:55:46,169 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-15 02:55:46,169 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34681,1681527209133' in zk 2023-04-15 02:55:46,169 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:46,169 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-15 02:55:46,170 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:46,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:46,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:55:46,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:55:46,170 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-15 02:55:46,170 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-15 02:55:46,170 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-15 02:55:46,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:55:46,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:55:46,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:55:46,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:46,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:55:46,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:55:46,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:46,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34681,1681527209133': 2023-04-15 02:55:46,172 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34681,1681527209133' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-15 02:55:46,172 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-15 02:55:46,172 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-15 02:55:46,172 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-15 02:55:46,172 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-15 02:55:46,172 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-15 02:55:46,173 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:46,173 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:46,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:46,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:55:46,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:55:46,173 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:46,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:46,173 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-15 02:55:46,173 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-15 02:55:46,173 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:46,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:55:46,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-15 02:55:46,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:55:46,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:46,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:55:46,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:55:46,174 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:46,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:46,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:55:46,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:55:46,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:46,175 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:46,175 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:55:46,175 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:55:46,175 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:46,175 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-15 02:55:46,175 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-15 02:55:46,175 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:46,175 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-15 02:55:46,176 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:46,176 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-15 02:55:46,176 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-15 02:55:46,176 DEBUG [Thread-2739] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-15 02:55:46,176 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:46,176 DEBUG [Thread-2739] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-15 02:55:46,176 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-15 02:55:46,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-15 02:55:46,176 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-15 02:55:46,176 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-15 02:55:46,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-15 02:55:46,276 DEBUG [Thread-2739] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-15 02:55:46,276 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-15 02:55:46,278 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-15 02:55:46,278 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-15 02:55:46,278 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-15 02:55:46,278 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-15 02:55:46,278 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-15 02:55:46,279 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-15 02:55:46,279 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-15 02:55:46,279 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:46,279 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:55:46,280 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:46,280 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-15 02:55:46,280 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-15 02:55:46,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-15 02:55:46,281 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:46,281 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-15 02:55:46,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:55:46,281 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:46,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-15 02:55:46,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:55:46,281 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-15 02:55:46,281 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-15 02:55:46,281 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-15 02:55:46,281 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-15 02:55:46,281 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-15 02:55:46,282 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-15 02:55:46,282 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-15 02:55:46,282 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-15 02:55:46,282 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681527331158.a29fa882a9e3f50c391e3aa08c5c5e72. 2023-04-15 02:55:46,282 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681527331158.a29fa882a9e3f50c391e3aa08c5c5e72. started... 2023-04-15 02:55:46,282 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681527331158.a29fa882a9e3f50c391e3aa08c5c5e72. as already flushing 2023-04-15 02:55:46,282 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681527331158.a29fa882a9e3f50c391e3aa08c5c5e72. 2023-04-15 02:55:46,282 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-15 02:55:46,282 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-15 02:55:46,282 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:46,282 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-15 02:55:46,282 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34681,1681527209133' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-15 02:55:46,283 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:46,283 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:46,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:46,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:55:46,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:55:46,283 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:46,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:55:46,283 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-15 02:55:46,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:55:46,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:55:46,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:46,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:55:46,284 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34681,1681527209133' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-15 02:55:46,284 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1e187b89[Count = 0] remaining members to acquire global barrier 2023-04-15 02:55:46,284 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-15 02:55:46,284 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:46,285 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:46,285 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:46,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:46,285 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-15 02:55:46,285 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-15 02:55:46,285 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:46,285 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-15 02:55:46,285 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34681,1681527209133' in zk 2023-04-15 02:55:46,286 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:46,286 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-15 02:55:46,286 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-15 02:55:46,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:46,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:55:46,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:55:46,286 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-15 02:55:46,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:55:46,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:55:46,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:55:46,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:46,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:55:46,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:55:46,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:46,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34681,1681527209133': 2023-04-15 02:55:46,287 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34681,1681527209133' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-15 02:55:46,287 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-15 02:55:46,287 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-15 02:55:46,287 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-15 02:55:46,287 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-15 02:55:46,287 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-15 02:55:46,288 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:46,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:46,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:55:46,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:55:46,288 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:46,288 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-15 02:55:46,288 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:46,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:46,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:55:46,289 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:46,289 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-15 02:55:46,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-15 02:55:46,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:55:46,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:46,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:55:46,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:55:46,289 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:46,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:46,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:55:46,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:55:46,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:46,291 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:46,291 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:55:46,291 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:55:46,291 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-15 02:55:46,291 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-15 02:55:46,291 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-15 02:55:46,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-15 02:55:46,291 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-15 02:55:46,291 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:46,291 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-15 02:55:46,291 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-15 02:55:46,291 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-15 02:55:46,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-15 02:55:46,291 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:46,292 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:46,292 DEBUG [Thread-2739] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-15 02:55:46,292 DEBUG [Thread-2739] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-15 02:55:46,292 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:46,366 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a29fa882a9e3f50c391e3aa08c5c5e72, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:55:46,366 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34681] ipc.CallRunner(144): callId: 168 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:40234 deadline: 1681527406361, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a29fa882a9e3f50c391e3aa08c5c5e72, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:46,367 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a29fa882a9e3f50c391e3aa08c5c5e72, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:55:46,367 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34681] ipc.CallRunner(144): callId: 172 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:40210 deadline: 1681527406362, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a29fa882a9e3f50c391e3aa08c5c5e72, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:46,373 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a29fa882a9e3f50c391e3aa08c5c5e72, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:55:46,373 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34681] ipc.CallRunner(144): callId: 171 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:40192 deadline: 1681527406367, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a29fa882a9e3f50c391e3aa08c5c5e72, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:46,392 DEBUG [Thread-2739] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-15 02:55:46,392 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-15 02:55:46,393 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-15 02:55:46,394 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-15 02:55:46,394 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-15 02:55:46,394 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-15 02:55:46,394 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-15 02:55:46,394 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-15 02:55:46,394 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-15 02:55:46,395 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:46,395 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:55:46,395 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:46,395 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-15 02:55:46,395 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-15 02:55:46,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-15 02:55:46,396 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:46,396 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-15 02:55:46,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:55:46,396 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:46,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-15 02:55:46,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:55:46,396 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-15 02:55:46,396 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-15 02:55:46,396 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-15 02:55:46,397 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-15 02:55:46,397 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-15 02:55:46,397 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-15 02:55:46,397 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-15 02:55:46,397 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-15 02:55:46,397 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681527331158.a29fa882a9e3f50c391e3aa08c5c5e72. 2023-04-15 02:55:46,397 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681527331158.a29fa882a9e3f50c391e3aa08c5c5e72. started... 2023-04-15 02:55:46,397 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681527331158.a29fa882a9e3f50c391e3aa08c5c5e72. as already flushing 2023-04-15 02:55:46,397 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681527331158.a29fa882a9e3f50c391e3aa08c5c5e72. 2023-04-15 02:55:46,397 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-15 02:55:46,397 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-15 02:55:46,397 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:46,397 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-15 02:55:46,397 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34681,1681527209133' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-15 02:55:46,398 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:46,398 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:46,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:46,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:55:46,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:55:46,398 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:46,398 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-15 02:55:46,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:55:46,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:55:46,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:55:46,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:46,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:55:46,399 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34681,1681527209133' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-15 02:55:46,399 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-15 02:55:46,399 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2f601e4a[Count = 0] remaining members to acquire global barrier 2023-04-15 02:55:46,399 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:46,400 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:46,400 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:46,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:46,400 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-15 02:55:46,400 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-15 02:55:46,400 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34681,1681527209133' in zk 2023-04-15 02:55:46,400 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:46,400 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-15 02:55:46,400 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:46,400 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-15 02:55:46,401 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-15 02:55:46,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:46,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:55:46,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:55:46,401 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-15 02:55:46,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:55:46,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:55:46,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:55:46,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:46,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:55:46,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:55:46,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:46,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34681,1681527209133': 2023-04-15 02:55:46,402 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34681,1681527209133' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-15 02:55:46,402 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-15 02:55:46,402 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-15 02:55:46,402 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-15 02:55:46,402 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-15 02:55:46,402 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-15 02:55:46,403 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:46,403 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-15 02:55:46,403 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:46,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:46,403 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:46,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:46,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:55:46,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:55:46,404 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-15 02:55:46,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-15 02:55:46,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:55:46,404 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:46,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:46,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:55:46,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:55:46,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:55:46,405 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:46,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:46,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:55:46,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:55:46,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:46,406 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:46,406 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:55:46,406 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-15 02:55:46,406 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-15 02:55:46,406 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-15 02:55:46,406 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-15 02:55:46,406 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-15 02:55:46,406 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:55:46,406 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-15 02:55:46,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-15 02:55:46,406 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:46,406 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-15 02:55:46,406 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:46,406 DEBUG [Thread-2739] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-15 02:55:46,406 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:46,406 DEBUG [Thread-2739] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-15 02:55:46,406 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:46,406 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-15 02:55:46,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-15 02:55:46,501 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=323 (bloomFilter=true), to=hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/a29fa882a9e3f50c391e3aa08c5c5e72/.tmp/C/14944fc722fc45eba9b007613666b434 2023-04-15 02:55:46,506 DEBUG [Thread-2739] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-15 02:55:46,507 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-15 02:55:46,507 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/a29fa882a9e3f50c391e3aa08c5c5e72/.tmp/A/cb1cebeccfd7464b84d6eb0a083c50c0 as hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/a29fa882a9e3f50c391e3aa08c5c5e72/A/cb1cebeccfd7464b84d6eb0a083c50c0 2023-04-15 02:55:46,508 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-15 02:55:46,509 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-15 02:55:46,509 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-15 02:55:46,509 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-15 02:55:46,509 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-15 02:55:46,509 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-15 02:55:46,509 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-15 02:55:46,509 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:46,509 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:55:46,510 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-15 02:55:46,510 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/a29fa882a9e3f50c391e3aa08c5c5e72/A/cb1cebeccfd7464b84d6eb0a083c50c0, entries=200, sequenceid=323, filesize=14.3 K 2023-04-15 02:55:46,510 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-15 02:55:46,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-15 02:55:46,510 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:46,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:55:46,510 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:46,511 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-15 02:55:46,511 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:46,511 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/a29fa882a9e3f50c391e3aa08c5c5e72/.tmp/B/1f7949a7a669426ab732be2e19ad4f6d as hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/a29fa882a9e3f50c391e3aa08c5c5e72/B/1f7949a7a669426ab732be2e19ad4f6d 2023-04-15 02:55:46,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-15 02:55:46,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:55:46,511 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-15 02:55:46,511 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-15 02:55:46,511 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-15 02:55:46,511 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-15 02:55:46,511 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-15 02:55:46,511 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-15 02:55:46,512 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-15 02:55:46,512 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-15 02:55:46,512 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681527331158.a29fa882a9e3f50c391e3aa08c5c5e72. 2023-04-15 02:55:46,512 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681527331158.a29fa882a9e3f50c391e3aa08c5c5e72. started... 2023-04-15 02:55:46,512 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681527331158.a29fa882a9e3f50c391e3aa08c5c5e72. as already flushing 2023-04-15 02:55:46,512 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681527331158.a29fa882a9e3f50c391e3aa08c5c5e72. 2023-04-15 02:55:46,512 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-15 02:55:46,512 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-15 02:55:46,512 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:46,512 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-15 02:55:46,512 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34681,1681527209133' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-15 02:55:46,513 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:46,513 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:46,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:46,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:55:46,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:55:46,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:55:46,513 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:46,513 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-15 02:55:46,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:55:46,514 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/a29fa882a9e3f50c391e3aa08c5c5e72/B/1f7949a7a669426ab732be2e19ad4f6d, entries=150, sequenceid=323, filesize=11.9 K 2023-04-15 02:55:46,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:55:46,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:46,514 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/a29fa882a9e3f50c391e3aa08c5c5e72/.tmp/C/14944fc722fc45eba9b007613666b434 as hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/a29fa882a9e3f50c391e3aa08c5c5e72/C/14944fc722fc45eba9b007613666b434 2023-04-15 02:55:46,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:55:46,515 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34681,1681527209133' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-15 02:55:46,515 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@74453204[Count = 0] remaining members to acquire global barrier 2023-04-15 02:55:46,515 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-15 02:55:46,515 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:46,515 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:46,515 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:46,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:46,515 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-15 02:55:46,515 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-15 02:55:46,516 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34681,1681527209133' in zk 2023-04-15 02:55:46,516 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:46,516 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-15 02:55:46,516 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:46,516 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-15 02:55:46,516 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-15 02:55:46,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:46,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:55:46,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:55:46,516 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-15 02:55:46,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:55:46,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:55:46,517 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/a29fa882a9e3f50c391e3aa08c5c5e72/C/14944fc722fc45eba9b007613666b434, entries=150, sequenceid=323, filesize=11.9 K 2023-04-15 02:55:46,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:55:46,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:46,517 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=134.18 KB/137400 for a29fa882a9e3f50c391e3aa08c5c5e72 in 840ms, sequenceid=323, compaction requested=true 2023-04-15 02:55:46,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:55:46,518 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for a29fa882a9e3f50c391e3aa08c5c5e72: 2023-04-15 02:55:46,518 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-15 02:55:46,518 DEBUG [RS:0;jenkins-hbase20:34681-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-15 02:55:46,518 DEBUG [RS:0;jenkins-hbase20:34681-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-15 02:55:46,518 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-15 02:55:46,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:55:46,518 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-15 02:55:46,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:46,518 DEBUG [RS:0;jenkins-hbase20:34681-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 39578 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-15 02:55:46,518 DEBUG [RS:0;jenkins-hbase20:34681-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37138 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-15 02:55:46,519 DEBUG [RS:0;jenkins-hbase20:34681-shortCompactions-0] regionserver.HStore(1912): a29fa882a9e3f50c391e3aa08c5c5e72/A is initiating minor compaction (all files) 2023-04-15 02:55:46,519 DEBUG [RS:0;jenkins-hbase20:34681-longCompactions-0] regionserver.HStore(1912): a29fa882a9e3f50c391e3aa08c5c5e72/B is initiating minor compaction (all files) 2023-04-15 02:55:46,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34681,1681527209133': 2023-04-15 02:55:46,519 INFO [RS:0;jenkins-hbase20:34681-longCompactions-0] regionserver.HRegion(2259): Starting compaction of a29fa882a9e3f50c391e3aa08c5c5e72/B in TestAcidGuarantees,,1681527331158.a29fa882a9e3f50c391e3aa08c5c5e72. 2023-04-15 02:55:46,519 INFO [RS:0;jenkins-hbase20:34681-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of a29fa882a9e3f50c391e3aa08c5c5e72/A in TestAcidGuarantees,,1681527331158.a29fa882a9e3f50c391e3aa08c5c5e72. 2023-04-15 02:55:46,519 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-15 02:55:46,519 INFO [RS:0;jenkins-hbase20:34681-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/a29fa882a9e3f50c391e3aa08c5c5e72/A/cdcb1334de5b4aceb8a27f5d37d7cc43, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/a29fa882a9e3f50c391e3aa08c5c5e72/A/1e6fdf63e00641cf974a6d51d0c9c42f, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/a29fa882a9e3f50c391e3aa08c5c5e72/A/cb1cebeccfd7464b84d6eb0a083c50c0] into tmpdir=hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/a29fa882a9e3f50c391e3aa08c5c5e72/.tmp, totalSize=38.7 K 2023-04-15 02:55:46,519 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34681,1681527209133' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-15 02:55:46,519 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-15 02:55:46,519 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-15 02:55:46,519 INFO [RS:0;jenkins-hbase20:34681-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/a29fa882a9e3f50c391e3aa08c5c5e72/B/7876501289af46cfbf6bfb8b5d2d89f2, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/a29fa882a9e3f50c391e3aa08c5c5e72/B/47c3b4716c284989b651f97044b9111d, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/a29fa882a9e3f50c391e3aa08c5c5e72/B/1f7949a7a669426ab732be2e19ad4f6d] into tmpdir=hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/a29fa882a9e3f50c391e3aa08c5c5e72/.tmp, totalSize=36.3 K 2023-04-15 02:55:46,519 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-15 02:55:46,519 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-15 02:55:46,519 DEBUG [RS:0;jenkins-hbase20:34681-shortCompactions-0] compactions.Compactor(207): Compacting cdcb1334de5b4aceb8a27f5d37d7cc43, keycount=150, bloomtype=ROW, size=12.5 K, encoding=NONE, compression=NONE, seqNum=271, earliestPutTs=1681527343457 2023-04-15 02:55:46,519 DEBUG [RS:0;jenkins-hbase20:34681-longCompactions-0] compactions.Compactor(207): Compacting 7876501289af46cfbf6bfb8b5d2d89f2, keycount=150, bloomtype=ROW, size=12.5 K, encoding=NONE, compression=NONE, seqNum=271, earliestPutTs=1681527343457 2023-04-15 02:55:46,519 DEBUG [RS:0;jenkins-hbase20:34681-shortCompactions-0] compactions.Compactor(207): Compacting 1e6fdf63e00641cf974a6d51d0c9c42f, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=307, earliestPutTs=1681527343515 2023-04-15 02:55:46,520 DEBUG [RS:0;jenkins-hbase20:34681-longCompactions-0] compactions.Compactor(207): Compacting 47c3b4716c284989b651f97044b9111d, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=307, earliestPutTs=1681527343515 2023-04-15 02:55:46,520 DEBUG [RS:0;jenkins-hbase20:34681-shortCompactions-0] compactions.Compactor(207): Compacting cb1cebeccfd7464b84d6eb0a083c50c0, keycount=200, bloomtype=ROW, size=14.3 K, encoding=NONE, compression=NONE, seqNum=323, earliestPutTs=1681527345668 2023-04-15 02:55:46,520 DEBUG [RS:0;jenkins-hbase20:34681-longCompactions-0] compactions.Compactor(207): Compacting 1f7949a7a669426ab732be2e19ad4f6d, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=323, earliestPutTs=1681527345671 2023-04-15 02:55:46,520 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:46,520 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-15 02:55:46,520 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:46,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:46,521 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:46,522 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:46,522 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-15 02:55:46,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-15 02:55:46,522 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:46,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:46,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:46,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:55:46,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:55:46,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:55:46,524 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-15 02:55:46,524 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-15 02:55:46,524 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-15 02:55:46,524 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-15 02:55:46,524 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-15 02:55:46,524 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-15 02:55:46,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:55:46,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-15 02:55:46,524 DEBUG [Thread-2739] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-15 02:55:46,524 DEBUG [Thread-2739] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-15 02:55:46,524 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-15 02:55:46,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-15 02:55:46,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:55:46,525 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:46,525 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:55:46,525 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:55:46,525 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:46,525 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:46,525 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:46,525 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:46,525 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-15 02:55:46,527 INFO [RS:0;jenkins-hbase20:34681-shortCompactions-0] throttle.PressureAwareThroughputController(145): a29fa882a9e3f50c391e3aa08c5c5e72#A#compaction#405 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-15 02:55:46,527 INFO [RS:0;jenkins-hbase20:34681-longCompactions-0] throttle.PressureAwareThroughputController(145): a29fa882a9e3f50c391e3aa08c5c5e72#B#compaction#404 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-15 02:55:46,534 DEBUG [RS:0;jenkins-hbase20:34681-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/a29fa882a9e3f50c391e3aa08c5c5e72/.tmp/A/1ee1d258316345379a04f4b9bd87e5ed as hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/a29fa882a9e3f50c391e3aa08c5c5e72/A/1ee1d258316345379a04f4b9bd87e5ed 2023-04-15 02:55:46,535 DEBUG [RS:0;jenkins-hbase20:34681-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/a29fa882a9e3f50c391e3aa08c5c5e72/.tmp/B/f6e80b7413c74242b9c7c04ee2582e18 as hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/a29fa882a9e3f50c391e3aa08c5c5e72/B/f6e80b7413c74242b9c7c04ee2582e18 2023-04-15 02:55:46,538 INFO [RS:0;jenkins-hbase20:34681-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in a29fa882a9e3f50c391e3aa08c5c5e72/A of a29fa882a9e3f50c391e3aa08c5c5e72 into 1ee1d258316345379a04f4b9bd87e5ed(size=12.6 K), total size for store is 12.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-15 02:55:46,538 DEBUG [RS:0;jenkins-hbase20:34681-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for a29fa882a9e3f50c391e3aa08c5c5e72: 2023-04-15 02:55:46,538 INFO [RS:0;jenkins-hbase20:34681-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681527331158.a29fa882a9e3f50c391e3aa08c5c5e72., storeName=a29fa882a9e3f50c391e3aa08c5c5e72/A, priority=13, startTime=1681527346518; duration=0sec 2023-04-15 02:55:46,538 INFO [RS:0;jenkins-hbase20:34681-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in a29fa882a9e3f50c391e3aa08c5c5e72/B of a29fa882a9e3f50c391e3aa08c5c5e72 into f6e80b7413c74242b9c7c04ee2582e18(size=12.6 K), total size for store is 12.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-15 02:55:46,538 DEBUG [RS:0;jenkins-hbase20:34681-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for a29fa882a9e3f50c391e3aa08c5c5e72: 2023-04-15 02:55:46,538 DEBUG [RS:0;jenkins-hbase20:34681-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-15 02:55:46,538 INFO [RS:0;jenkins-hbase20:34681-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681527331158.a29fa882a9e3f50c391e3aa08c5c5e72., storeName=a29fa882a9e3f50c391e3aa08c5c5e72/B, priority=13, startTime=1681527346518; duration=0sec 2023-04-15 02:55:46,538 DEBUG [RS:0;jenkins-hbase20:34681-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-15 02:55:46,538 DEBUG [RS:0;jenkins-hbase20:34681-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-15 02:55:46,539 DEBUG [RS:0;jenkins-hbase20:34681-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37138 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-15 02:55:46,539 DEBUG [RS:0;jenkins-hbase20:34681-shortCompactions-0] regionserver.HStore(1912): a29fa882a9e3f50c391e3aa08c5c5e72/C is initiating minor compaction (all files) 2023-04-15 02:55:46,539 INFO [RS:0;jenkins-hbase20:34681-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of a29fa882a9e3f50c391e3aa08c5c5e72/C in TestAcidGuarantees,,1681527331158.a29fa882a9e3f50c391e3aa08c5c5e72. 2023-04-15 02:55:46,539 INFO [RS:0;jenkins-hbase20:34681-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/a29fa882a9e3f50c391e3aa08c5c5e72/C/2a2de79b0f804d0aa7c763bfa19f47b1, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/a29fa882a9e3f50c391e3aa08c5c5e72/C/8e9b2ace06874a618b994b9577a16160, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/a29fa882a9e3f50c391e3aa08c5c5e72/C/14944fc722fc45eba9b007613666b434] into tmpdir=hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/a29fa882a9e3f50c391e3aa08c5c5e72/.tmp, totalSize=36.3 K 2023-04-15 02:55:46,539 DEBUG [RS:0;jenkins-hbase20:34681-shortCompactions-0] compactions.Compactor(207): Compacting 2a2de79b0f804d0aa7c763bfa19f47b1, keycount=150, bloomtype=ROW, size=12.5 K, encoding=NONE, compression=NONE, seqNum=271, earliestPutTs=1681527343457 2023-04-15 02:55:46,540 DEBUG [RS:0;jenkins-hbase20:34681-shortCompactions-0] compactions.Compactor(207): Compacting 8e9b2ace06874a618b994b9577a16160, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=307, earliestPutTs=1681527343515 2023-04-15 02:55:46,540 DEBUG [RS:0;jenkins-hbase20:34681-shortCompactions-0] compactions.Compactor(207): Compacting 14944fc722fc45eba9b007613666b434, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=323, earliestPutTs=1681527345671 2023-04-15 02:55:46,545 INFO [RS:0;jenkins-hbase20:34681-shortCompactions-0] throttle.PressureAwareThroughputController(145): a29fa882a9e3f50c391e3aa08c5c5e72#C#compaction#406 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-15 02:55:46,624 DEBUG [Thread-2739] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-15 02:55:46,625 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-15 02:55:46,626 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-15 02:55:46,627 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-15 02:55:46,627 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-15 02:55:46,627 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-15 02:55:46,627 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-15 02:55:46,627 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-15 02:55:46,627 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-15 02:55:46,627 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:46,627 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:55:46,628 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-15 02:55:46,628 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:46,628 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-15 02:55:46,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-15 02:55:46,628 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:46,628 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-15 02:55:46,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:55:46,628 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:46,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-15 02:55:46,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:55:46,629 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-15 02:55:46,629 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-15 02:55:46,629 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-15 02:55:46,629 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-15 02:55:46,629 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-15 02:55:46,629 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-15 02:55:46,629 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-15 02:55:46,629 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-15 02:55:46,629 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681527331158.a29fa882a9e3f50c391e3aa08c5c5e72. 2023-04-15 02:55:46,629 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681527331158.a29fa882a9e3f50c391e3aa08c5c5e72. started... 2023-04-15 02:55:46,629 INFO [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing a29fa882a9e3f50c391e3aa08c5c5e72 3/3 column families, dataSize=134.18 KB heapSize=380.30 KB 2023-04-15 02:55:46,630 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a29fa882a9e3f50c391e3aa08c5c5e72, store=A 2023-04-15 02:55:46,630 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-15 02:55:46,630 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a29fa882a9e3f50c391e3aa08c5c5e72, store=B 2023-04-15 02:55:46,630 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-15 02:55:46,630 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a29fa882a9e3f50c391e3aa08c5c5e72, store=C 2023-04-15 02:55:46,630 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-15 02:55:46,636 INFO [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=361 (bloomFilter=true), to=hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/a29fa882a9e3f50c391e3aa08c5c5e72/.tmp/A/e92ab01b17d04e6692817e2ec14e1214 2023-04-15 02:55:46,870 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34681] regionserver.HRegion(9158): Flush requested on a29fa882a9e3f50c391e3aa08c5c5e72 2023-04-15 02:55:46,870 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681527331158.a29fa882a9e3f50c391e3aa08c5c5e72. as already flushing 2023-04-15 02:55:46,885 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a29fa882a9e3f50c391e3aa08c5c5e72, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:55:46,886 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34681] ipc.CallRunner(144): callId: 173 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:40234 deadline: 1681527406880, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a29fa882a9e3f50c391e3aa08c5c5e72, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:46,886 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a29fa882a9e3f50c391e3aa08c5c5e72, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:55:46,886 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34681] ipc.CallRunner(144): callId: 174 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:40192 deadline: 1681527406880, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a29fa882a9e3f50c391e3aa08c5c5e72, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:46,886 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a29fa882a9e3f50c391e3aa08c5c5e72, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:55:46,886 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34681] ipc.CallRunner(144): callId: 177 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:40210 deadline: 1681527406881, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a29fa882a9e3f50c391e3aa08c5c5e72, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:46,951 DEBUG [RS:0;jenkins-hbase20:34681-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/a29fa882a9e3f50c391e3aa08c5c5e72/.tmp/C/0bb7d65eea50491994aff94d3a8396a5 as hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/a29fa882a9e3f50c391e3aa08c5c5e72/C/0bb7d65eea50491994aff94d3a8396a5 2023-04-15 02:55:46,955 INFO [RS:0;jenkins-hbase20:34681-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in a29fa882a9e3f50c391e3aa08c5c5e72/C of a29fa882a9e3f50c391e3aa08c5c5e72 into 0bb7d65eea50491994aff94d3a8396a5(size=12.6 K), total size for store is 12.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-15 02:55:46,955 DEBUG [RS:0;jenkins-hbase20:34681-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for a29fa882a9e3f50c391e3aa08c5c5e72: 2023-04-15 02:55:46,955 INFO [RS:0;jenkins-hbase20:34681-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681527331158.a29fa882a9e3f50c391e3aa08c5c5e72., storeName=a29fa882a9e3f50c391e3aa08c5c5e72/C, priority=13, startTime=1681527346518; duration=0sec 2023-04-15 02:55:46,955 DEBUG [RS:0;jenkins-hbase20:34681-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-15 02:55:46,988 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a29fa882a9e3f50c391e3aa08c5c5e72, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:55:46,988 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34681] ipc.CallRunner(144): callId: 175 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:40234 deadline: 1681527406986, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a29fa882a9e3f50c391e3aa08c5c5e72, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:46,988 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a29fa882a9e3f50c391e3aa08c5c5e72, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:55:46,988 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34681] ipc.CallRunner(144): callId: 176 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:40192 deadline: 1681527406987, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a29fa882a9e3f50c391e3aa08c5c5e72, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:46,988 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a29fa882a9e3f50c391e3aa08c5c5e72, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:55:46,988 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34681] ipc.CallRunner(144): callId: 179 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:40210 deadline: 1681527406987, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a29fa882a9e3f50c391e3aa08c5c5e72, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:47,053 INFO [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=361 (bloomFilter=true), to=hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/a29fa882a9e3f50c391e3aa08c5c5e72/.tmp/B/6ba49c87f56a4a3cb7b3c3b548430ce1 2023-04-15 02:55:47,191 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a29fa882a9e3f50c391e3aa08c5c5e72, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:55:47,192 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34681] ipc.CallRunner(144): callId: 181 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:40210 deadline: 1681527407189, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a29fa882a9e3f50c391e3aa08c5c5e72, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:47,192 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a29fa882a9e3f50c391e3aa08c5c5e72, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:55:47,192 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34681] ipc.CallRunner(144): callId: 178 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:40192 deadline: 1681527407190, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a29fa882a9e3f50c391e3aa08c5c5e72, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:47,192 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a29fa882a9e3f50c391e3aa08c5c5e72, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:55:47,192 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34681] ipc.CallRunner(144): callId: 177 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:40234 deadline: 1681527407190, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a29fa882a9e3f50c391e3aa08c5c5e72, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:47,462 INFO [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=361 (bloomFilter=true), to=hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/a29fa882a9e3f50c391e3aa08c5c5e72/.tmp/C/49455a4d121741a59906287bbd6150b2 2023-04-15 02:55:47,465 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/a29fa882a9e3f50c391e3aa08c5c5e72/.tmp/A/e92ab01b17d04e6692817e2ec14e1214 as hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/a29fa882a9e3f50c391e3aa08c5c5e72/A/e92ab01b17d04e6692817e2ec14e1214 2023-04-15 02:55:47,467 INFO [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/a29fa882a9e3f50c391e3aa08c5c5e72/A/e92ab01b17d04e6692817e2ec14e1214, entries=150, sequenceid=361, filesize=11.9 K 2023-04-15 02:55:47,468 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/a29fa882a9e3f50c391e3aa08c5c5e72/.tmp/B/6ba49c87f56a4a3cb7b3c3b548430ce1 as hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/a29fa882a9e3f50c391e3aa08c5c5e72/B/6ba49c87f56a4a3cb7b3c3b548430ce1 2023-04-15 02:55:47,471 INFO [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/a29fa882a9e3f50c391e3aa08c5c5e72/B/6ba49c87f56a4a3cb7b3c3b548430ce1, entries=150, sequenceid=361, filesize=11.9 K 2023-04-15 02:55:47,471 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/a29fa882a9e3f50c391e3aa08c5c5e72/.tmp/C/49455a4d121741a59906287bbd6150b2 as hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/a29fa882a9e3f50c391e3aa08c5c5e72/C/49455a4d121741a59906287bbd6150b2 2023-04-15 02:55:47,474 INFO [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/a29fa882a9e3f50c391e3aa08c5c5e72/C/49455a4d121741a59906287bbd6150b2, entries=150, sequenceid=361, filesize=11.9 K 2023-04-15 02:55:47,475 INFO [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~134.18 KB/137400, heapSize ~380.25 KB/389376, currentSize=46.96 KB/48090 for a29fa882a9e3f50c391e3aa08c5c5e72 in 846ms, sequenceid=361, compaction requested=false 2023-04-15 02:55:47,475 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for a29fa882a9e3f50c391e3aa08c5c5e72: 2023-04-15 02:55:47,475 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681527331158.a29fa882a9e3f50c391e3aa08c5c5e72. 2023-04-15 02:55:47,475 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-15 02:55:47,475 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-15 02:55:47,475 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:47,475 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-15 02:55:47,475 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34681,1681527209133' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-15 02:55:47,477 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:47,477 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:47,477 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-15 02:55:47,480 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:47,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:47,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:55:47,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:55:47,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:55:47,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:55:47,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:55:47,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:47,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:55:47,482 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34681,1681527209133' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-15 02:55:47,482 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@299e6bf6[Count = 0] remaining members to acquire global barrier 2023-04-15 02:55:47,482 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-15 02:55:47,482 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:47,488 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:47,488 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:47,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:47,488 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-15 02:55:47,488 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-15 02:55:47,488 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:47,488 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-15 02:55:47,488 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34681,1681527209133' in zk 2023-04-15 02:55:47,489 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:47,489 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-15 02:55:47,489 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-15 02:55:47,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:47,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:55:47,489 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-15 02:55:47,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:55:47,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:55:47,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:55:47,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:55:47,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:47,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:55:47,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:55:47,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:47,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34681,1681527209133': 2023-04-15 02:55:47,491 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34681,1681527209133' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-15 02:55:47,491 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-15 02:55:47,491 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-15 02:55:47,491 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-15 02:55:47,491 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-15 02:55:47,491 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-15 02:55:47,491 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:47,491 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:47,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:47,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:55:47,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:55:47,492 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:47,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:47,492 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-15 02:55:47,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:55:47,492 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:47,492 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-15 02:55:47,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-15 02:55:47,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:55:47,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:47,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:55:47,493 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:47,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:55:47,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:47,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:55:47,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:55:47,496 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-15 02:55:47,496 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:47,496 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-15 02:55:47,496 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-15 02:55:47,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-15 02:55:47,496 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34681] regionserver.HRegion(9158): Flush requested on a29fa882a9e3f50c391e3aa08c5c5e72 2023-04-15 02:55:47,496 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:55:47,496 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:55:47,496 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:47,496 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-15 02:55:47,496 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:47,496 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:47,496 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:47,496 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing a29fa882a9e3f50c391e3aa08c5c5e72 3/3 column families, dataSize=53.67 KB heapSize=160.13 KB 2023-04-15 02:55:47,496 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-15 02:55:47,496 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-15 02:55:47,496 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-15 02:55:47,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-15 02:55:47,496 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-15 02:55:47,496 DEBUG [Thread-2739] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-15 02:55:47,497 DEBUG [Thread-2739] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-15 02:55:47,497 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a29fa882a9e3f50c391e3aa08c5c5e72, store=A 2023-04-15 02:55:47,497 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-15 02:55:47,497 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a29fa882a9e3f50c391e3aa08c5c5e72, store=B 2023-04-15 02:55:47,497 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-15 02:55:47,497 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a29fa882a9e3f50c391e3aa08c5c5e72, store=C 2023-04-15 02:55:47,497 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-15 02:55:47,531 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a29fa882a9e3f50c391e3aa08c5c5e72, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:55:47,531 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34681] ipc.CallRunner(144): callId: 185 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:40234 deadline: 1681527407525, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a29fa882a9e3f50c391e3aa08c5c5e72, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:47,535 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a29fa882a9e3f50c391e3aa08c5c5e72, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:55:47,535 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34681] ipc.CallRunner(144): callId: 189 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:40210 deadline: 1681527407530, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a29fa882a9e3f50c391e3aa08c5c5e72, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:47,536 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a29fa882a9e3f50c391e3aa08c5c5e72, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:55:47,536 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34681] ipc.CallRunner(144): callId: 187 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:40192 deadline: 1681527407531, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a29fa882a9e3f50c391e3aa08c5c5e72, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:47,597 DEBUG [Thread-2739] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-15 02:55:47,597 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-15 02:55:47,598 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-15 02:55:47,599 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-15 02:55:47,599 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-15 02:55:47,599 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-15 02:55:47,599 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-15 02:55:47,599 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-15 02:55:47,599 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-15 02:55:47,600 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:47,600 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:55:47,600 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:47,600 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-15 02:55:47,600 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-15 02:55:47,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-15 02:55:47,601 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:47,601 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-15 02:55:47,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:55:47,601 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:47,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-15 02:55:47,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:55:47,601 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-15 02:55:47,601 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-15 02:55:47,601 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-15 02:55:47,601 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-15 02:55:47,601 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-15 02:55:47,601 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-15 02:55:47,602 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-15 02:55:47,602 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-15 02:55:47,602 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681527331158.a29fa882a9e3f50c391e3aa08c5c5e72. 2023-04-15 02:55:47,602 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681527331158.a29fa882a9e3f50c391e3aa08c5c5e72. started... 2023-04-15 02:55:47,602 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681527331158.a29fa882a9e3f50c391e3aa08c5c5e72. as already flushing 2023-04-15 02:55:47,602 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681527331158.a29fa882a9e3f50c391e3aa08c5c5e72. 2023-04-15 02:55:47,602 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-15 02:55:47,602 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-15 02:55:47,602 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:47,602 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-15 02:55:47,602 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34681,1681527209133' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-15 02:55:47,603 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:47,603 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:47,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:47,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:55:47,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:55:47,603 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:47,603 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-15 02:55:47,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:55:47,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:55:47,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:55:47,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:47,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:55:47,605 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34681,1681527209133' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-15 02:55:47,605 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7407897f[Count = 0] remaining members to acquire global barrier 2023-04-15 02:55:47,605 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-15 02:55:47,605 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:47,605 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:47,605 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:47,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:47,605 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-15 02:55:47,605 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-15 02:55:47,605 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34681,1681527209133' in zk 2023-04-15 02:55:47,605 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:47,605 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-15 02:55:47,606 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:47,606 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-15 02:55:47,606 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-15 02:55:47,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:47,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:55:47,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:55:47,606 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-15 02:55:47,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:55:47,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:55:47,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:55:47,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:47,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:55:47,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:55:47,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:47,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34681,1681527209133': 2023-04-15 02:55:47,608 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34681,1681527209133' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-15 02:55:47,608 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-15 02:55:47,608 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-15 02:55:47,608 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-15 02:55:47,608 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-15 02:55:47,608 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-15 02:55:47,609 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:47,609 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:47,609 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-15 02:55:47,609 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:47,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:47,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:47,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:55:47,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:55:47,609 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-15 02:55:47,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-15 02:55:47,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:47,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:55:47,610 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:47,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:55:47,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:55:47,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:55:47,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:47,611 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:47,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:55:47,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:55:47,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:47,612 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:47,613 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-15 02:55:47,613 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:55:47,613 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-15 02:55:47,613 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-15 02:55:47,613 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-15 02:55:47,613 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-15 02:55:47,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-15 02:55:47,613 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-15 02:55:47,613 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:55:47,613 DEBUG [Thread-2739] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-15 02:55:47,613 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:47,613 DEBUG [Thread-2739] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-15 02:55:47,613 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-15 02:55:47,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-15 02:55:47,613 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:47,613 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:47,613 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:47,635 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a29fa882a9e3f50c391e3aa08c5c5e72, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:55:47,635 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34681] ipc.CallRunner(144): callId: 187 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:40234 deadline: 1681527407632, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a29fa882a9e3f50c391e3aa08c5c5e72, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:47,639 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a29fa882a9e3f50c391e3aa08c5c5e72, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:55:47,639 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34681] ipc.CallRunner(144): callId: 191 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:40210 deadline: 1681527407636, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a29fa882a9e3f50c391e3aa08c5c5e72, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:47,640 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a29fa882a9e3f50c391e3aa08c5c5e72, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:55:47,640 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34681] ipc.CallRunner(144): callId: 189 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:40192 deadline: 1681527407637, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a29fa882a9e3f50c391e3aa08c5c5e72, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:47,713 DEBUG [Thread-2739] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-15 02:55:47,714 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-15 02:55:47,715 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-15 02:55:47,715 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-15 02:55:47,715 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-15 02:55:47,716 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-15 02:55:47,716 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-15 02:55:47,716 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-15 02:55:47,716 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-15 02:55:47,716 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:47,716 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:55:47,716 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-15 02:55:47,716 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:47,717 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-15 02:55:47,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-15 02:55:47,717 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:47,717 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-15 02:55:47,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:55:47,717 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:47,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-15 02:55:47,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:55:47,717 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-15 02:55:47,717 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-15 02:55:47,717 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-15 02:55:47,717 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-15 02:55:47,718 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-15 02:55:47,718 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-15 02:55:47,718 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-15 02:55:47,718 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-15 02:55:47,718 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681527331158.a29fa882a9e3f50c391e3aa08c5c5e72. 2023-04-15 02:55:47,718 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681527331158.a29fa882a9e3f50c391e3aa08c5c5e72. started... 2023-04-15 02:55:47,718 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681527331158.a29fa882a9e3f50c391e3aa08c5c5e72. as already flushing 2023-04-15 02:55:47,718 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681527331158.a29fa882a9e3f50c391e3aa08c5c5e72. 2023-04-15 02:55:47,718 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-15 02:55:47,718 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-15 02:55:47,718 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:47,718 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-15 02:55:47,718 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34681,1681527209133' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-15 02:55:47,719 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:47,719 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:47,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:47,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:55:47,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:55:47,719 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:47,719 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-15 02:55:47,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:55:47,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:55:47,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:55:47,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:47,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:55:47,720 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34681,1681527209133' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-15 02:55:47,720 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5b3a4bfb[Count = 0] remaining members to acquire global barrier 2023-04-15 02:55:47,720 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-15 02:55:47,720 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:47,720 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:47,720 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:47,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:47,720 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-15 02:55:47,720 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-15 02:55:47,720 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34681,1681527209133' in zk 2023-04-15 02:55:47,721 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-15 02:55:47,721 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-15 02:55:47,721 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-15 02:55:47,721 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:47,721 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-15 02:55:47,721 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:47,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:47,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:55:47,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:55:47,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:55:47,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:55:47,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:55:47,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:47,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:55:47,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:55:47,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:47,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34681,1681527209133': 2023-04-15 02:55:47,723 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34681,1681527209133' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-15 02:55:47,723 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-15 02:55:47,723 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-15 02:55:47,723 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-15 02:55:47,723 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-15 02:55:47,723 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-15 02:55:47,723 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:47,723 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:47,723 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-15 02:55:47,723 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:47,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:47,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:47,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:55:47,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:55:47,724 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-15 02:55:47,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-15 02:55:47,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:55:47,724 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:47,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:47,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:55:47,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:55:47,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:55:47,725 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:47,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:47,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:55:47,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:55:47,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:47,726 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:47,726 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-15 02:55:47,726 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:55:47,726 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-15 02:55:47,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-15 02:55:47,726 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-15 02:55:47,726 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-15 02:55:47,726 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-15 02:55:47,726 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:55:47,726 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-15 02:55:47,726 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-15 02:55:47,726 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-15 02:55:47,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-15 02:55:47,726 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:47,726 DEBUG [Thread-2739] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-15 02:55:47,726 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:47,726 DEBUG [Thread-2739] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-15 02:55:47,726 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:47,726 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:47,826 DEBUG [Thread-2739] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-15 02:55:47,827 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-15 02:55:47,828 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-15 02:55:47,829 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-15 02:55:47,829 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-15 02:55:47,829 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-15 02:55:47,829 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-15 02:55:47,829 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-15 02:55:47,829 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-15 02:55:47,829 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:47,829 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:55:47,830 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:47,830 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-15 02:55:47,830 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-15 02:55:47,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-15 02:55:47,830 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:47,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:55:47,830 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-15 02:55:47,831 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:47,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-15 02:55:47,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:55:47,831 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-15 02:55:47,831 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-15 02:55:47,831 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-15 02:55:47,831 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-15 02:55:47,831 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-15 02:55:47,831 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-15 02:55:47,831 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-15 02:55:47,831 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-15 02:55:47,831 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681527331158.a29fa882a9e3f50c391e3aa08c5c5e72. 2023-04-15 02:55:47,831 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681527331158.a29fa882a9e3f50c391e3aa08c5c5e72. started... 2023-04-15 02:55:47,832 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681527331158.a29fa882a9e3f50c391e3aa08c5c5e72. as already flushing 2023-04-15 02:55:47,832 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681527331158.a29fa882a9e3f50c391e3aa08c5c5e72. 2023-04-15 02:55:47,832 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-15 02:55:47,832 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-15 02:55:47,832 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:47,832 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-15 02:55:47,832 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34681,1681527209133' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-15 02:55:47,832 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:47,832 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:47,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:47,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:55:47,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:55:47,833 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:47,833 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-15 02:55:47,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:55:47,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:55:47,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:55:47,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:47,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:55:47,833 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34681,1681527209133' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-15 02:55:47,834 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6b2878a[Count = 0] remaining members to acquire global barrier 2023-04-15 02:55:47,834 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-15 02:55:47,834 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:47,834 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:47,834 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:47,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:47,834 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-15 02:55:47,834 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-15 02:55:47,834 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34681,1681527209133' in zk 2023-04-15 02:55:47,835 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:47,835 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-15 02:55:47,835 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:47,835 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-15 02:55:47,835 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-15 02:55:47,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:47,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:55:47,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:55:47,835 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-15 02:55:47,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:55:47,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:55:47,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:55:47,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:47,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:55:47,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:55:47,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:47,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34681,1681527209133': 2023-04-15 02:55:47,837 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34681,1681527209133' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-15 02:55:47,837 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-15 02:55:47,837 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-15 02:55:47,837 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-15 02:55:47,837 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-15 02:55:47,837 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-15 02:55:47,838 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:47,838 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:47,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:47,838 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:47,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:55:47,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:55:47,838 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-15 02:55:47,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:47,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:55:47,838 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-15 02:55:47,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-15 02:55:47,838 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:47,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:55:47,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:47,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:55:47,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:55:47,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:47,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:55:47,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:55:47,840 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:47,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:47,841 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:47,841 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-15 02:55:47,841 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:55:47,841 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-15 02:55:47,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-15 02:55:47,841 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-15 02:55:47,841 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-15 02:55:47,841 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-15 02:55:47,841 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-15 02:55:47,841 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:55:47,841 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-15 02:55:47,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-15 02:55:47,841 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:47,841 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:47,841 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:47,841 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:47,841 DEBUG [Thread-2739] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-15 02:55:47,842 DEBUG [Thread-2739] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-15 02:55:47,842 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a29fa882a9e3f50c391e3aa08c5c5e72, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:55:47,842 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34681] ipc.CallRunner(144): callId: 189 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:40234 deadline: 1681527407837, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a29fa882a9e3f50c391e3aa08c5c5e72, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:47,842 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a29fa882a9e3f50c391e3aa08c5c5e72, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:55:47,842 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34681] ipc.CallRunner(144): callId: 193 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:40210 deadline: 1681527407839, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a29fa882a9e3f50c391e3aa08c5c5e72, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:47,844 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a29fa882a9e3f50c391e3aa08c5c5e72, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:55:47,844 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34681] ipc.CallRunner(144): callId: 191 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:40192 deadline: 1681527407840, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a29fa882a9e3f50c391e3aa08c5c5e72, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:47,904 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=374 (bloomFilter=true), to=hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/a29fa882a9e3f50c391e3aa08c5c5e72/.tmp/A/6b6ea55ba4564b2885f935edea24a8df 2023-04-15 02:55:47,942 DEBUG [Thread-2739] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-15 02:55:47,942 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-15 02:55:47,944 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-15 02:55:47,944 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-15 02:55:47,944 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-15 02:55:47,944 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-15 02:55:47,944 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-15 02:55:47,944 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-15 02:55:47,945 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-15 02:55:47,945 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:47,945 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:55:47,946 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:47,946 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-15 02:55:47,946 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-15 02:55:47,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-15 02:55:47,946 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:47,946 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-15 02:55:47,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:55:47,946 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:47,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-15 02:55:47,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:55:47,946 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-15 02:55:47,946 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-15 02:55:47,947 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-15 02:55:47,947 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-15 02:55:47,947 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-15 02:55:47,947 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-15 02:55:47,947 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-15 02:55:47,947 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681527331158.a29fa882a9e3f50c391e3aa08c5c5e72. 2023-04-15 02:55:47,947 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681527331158.a29fa882a9e3f50c391e3aa08c5c5e72. started... 2023-04-15 02:55:47,947 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681527331158.a29fa882a9e3f50c391e3aa08c5c5e72. as already flushing 2023-04-15 02:55:47,947 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681527331158.a29fa882a9e3f50c391e3aa08c5c5e72. 2023-04-15 02:55:47,947 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-15 02:55:47,947 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-15 02:55:47,947 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:47,947 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-15 02:55:47,947 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34681,1681527209133' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-15 02:55:47,948 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:47,948 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:47,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:47,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:55:47,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:55:47,948 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:47,948 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-15 02:55:47,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:55:47,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:55:47,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:55:47,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:47,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:55:47,949 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34681,1681527209133' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-15 02:55:47,949 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-15 02:55:47,949 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5c299d71[Count = 0] remaining members to acquire global barrier 2023-04-15 02:55:47,949 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:47,950 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:47,950 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:47,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:47,950 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-15 02:55:47,950 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-15 02:55:47,950 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:47,950 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-15 02:55:47,950 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34681,1681527209133' in zk 2023-04-15 02:55:47,950 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-15 02:55:47,950 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-15 02:55:47,950 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:47,950 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-15 02:55:47,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:47,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:55:47,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:55:47,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:55:47,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:55:47,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:55:47,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:47,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:55:47,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:55:47,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:47,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34681,1681527209133': 2023-04-15 02:55:47,952 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34681,1681527209133' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-15 02:55:47,952 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-15 02:55:47,952 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-15 02:55:47,952 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-15 02:55:47,952 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-15 02:55:47,952 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-15 02:55:47,953 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:47,953 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:47,953 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-15 02:55:47,953 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:47,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:47,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:55:47,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:55:47,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:47,953 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-15 02:55:47,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:55:47,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-15 02:55:47,953 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:47,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:47,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:55:47,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:55:47,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:55:47,954 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:47,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:47,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:55:47,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:55:47,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:47,956 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:47,956 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-15 02:55:47,956 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:55:47,956 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-15 02:55:47,956 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-15 02:55:47,956 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-15 02:55:47,956 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-15 02:55:47,956 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-15 02:55:47,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-15 02:55:47,956 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-15 02:55:47,956 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-15 02:55:47,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-15 02:55:47,956 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:55:47,957 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:47,957 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:47,957 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:47,957 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:47,957 DEBUG [Thread-2739] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-15 02:55:47,957 DEBUG [Thread-2739] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-15 02:55:48,057 DEBUG [Thread-2739] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-15 02:55:48,058 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-15 02:55:48,059 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-15 02:55:48,060 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-15 02:55:48,060 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-15 02:55:48,060 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-15 02:55:48,060 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-15 02:55:48,060 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-15 02:55:48,060 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-15 02:55:48,060 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:48,060 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:55:48,061 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:48,061 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-15 02:55:48,061 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-15 02:55:48,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-15 02:55:48,061 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:48,061 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-15 02:55:48,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:55:48,062 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:48,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-15 02:55:48,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:55:48,062 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-15 02:55:48,062 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-15 02:55:48,062 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-15 02:55:48,062 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-15 02:55:48,062 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-15 02:55:48,062 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-15 02:55:48,062 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-15 02:55:48,062 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-15 02:55:48,062 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681527331158.a29fa882a9e3f50c391e3aa08c5c5e72. 2023-04-15 02:55:48,062 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681527331158.a29fa882a9e3f50c391e3aa08c5c5e72. started... 2023-04-15 02:55:48,063 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681527331158.a29fa882a9e3f50c391e3aa08c5c5e72. as already flushing 2023-04-15 02:55:48,063 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681527331158.a29fa882a9e3f50c391e3aa08c5c5e72. 2023-04-15 02:55:48,063 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-15 02:55:48,063 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-15 02:55:48,063 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:48,063 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-15 02:55:48,063 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34681,1681527209133' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-15 02:55:48,064 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:48,064 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:48,064 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-15 02:55:48,065 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:48,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:48,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:55:48,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:55:48,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:55:48,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:55:48,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:55:48,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:48,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:55:48,066 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34681,1681527209133' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-15 02:55:48,066 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-15 02:55:48,066 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@14adabfa[Count = 0] remaining members to acquire global barrier 2023-04-15 02:55:48,066 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:48,067 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:48,067 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:48,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:48,067 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-15 02:55:48,067 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-15 02:55:48,067 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:48,067 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-15 02:55:48,067 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34681,1681527209133' in zk 2023-04-15 02:55:48,067 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:48,067 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-15 02:55:48,067 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-15 02:55:48,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:48,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:55:48,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:55:48,068 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-15 02:55:48,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:55:48,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:55:48,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:55:48,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:48,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:55:48,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:55:48,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:48,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34681,1681527209133': 2023-04-15 02:55:48,070 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34681,1681527209133' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-15 02:55:48,070 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-15 02:55:48,070 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-15 02:55:48,070 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-15 02:55:48,070 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-15 02:55:48,070 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-15 02:55:48,071 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:48,071 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:48,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:48,071 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:48,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:55:48,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:55:48,071 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-15 02:55:48,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:48,071 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:48,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:55:48,071 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-15 02:55:48,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-15 02:55:48,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:48,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:55:48,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:55:48,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:55:48,072 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:48,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:48,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:55:48,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:55:48,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:48,073 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-15 02:55:48,073 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-15 02:55:48,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-15 02:55:48,073 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-15 02:55:48,073 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-15 02:55:48,073 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-15 02:55:48,073 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-15 02:55:48,073 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:48,073 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:55:48,073 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:55:48,073 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-15 02:55:48,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-15 02:55:48,073 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-15 02:55:48,073 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:48,073 DEBUG [Thread-2739] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-15 02:55:48,074 DEBUG [Thread-2739] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-15 02:55:48,074 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:48,074 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:48,074 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:48,146 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a29fa882a9e3f50c391e3aa08c5c5e72, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:55:48,146 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34681] ipc.CallRunner(144): callId: 191 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:40234 deadline: 1681527408143, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a29fa882a9e3f50c391e3aa08c5c5e72, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:48,149 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a29fa882a9e3f50c391e3aa08c5c5e72, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:55:48,149 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34681] ipc.CallRunner(144): callId: 195 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:40210 deadline: 1681527408146, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a29fa882a9e3f50c391e3aa08c5c5e72, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:48,150 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a29fa882a9e3f50c391e3aa08c5c5e72, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:55:48,150 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34681] ipc.CallRunner(144): callId: 193 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:40192 deadline: 1681527408146, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a29fa882a9e3f50c391e3aa08c5c5e72, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:48,174 DEBUG [Thread-2739] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-15 02:55:48,174 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-15 02:55:48,175 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-15 02:55:48,176 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-15 02:55:48,176 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-15 02:55:48,176 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-15 02:55:48,176 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-15 02:55:48,176 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-15 02:55:48,176 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-15 02:55:48,177 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:48,177 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:55:48,177 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:48,177 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-15 02:55:48,177 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-15 02:55:48,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-15 02:55:48,177 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:48,177 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-15 02:55:48,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:55:48,178 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:48,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-15 02:55:48,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:55:48,178 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-15 02:55:48,178 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-15 02:55:48,178 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-15 02:55:48,178 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-15 02:55:48,178 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-15 02:55:48,178 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-15 02:55:48,178 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-15 02:55:48,178 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-15 02:55:48,178 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681527331158.a29fa882a9e3f50c391e3aa08c5c5e72. 2023-04-15 02:55:48,179 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681527331158.a29fa882a9e3f50c391e3aa08c5c5e72. started... 2023-04-15 02:55:48,179 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681527331158.a29fa882a9e3f50c391e3aa08c5c5e72. as already flushing 2023-04-15 02:55:48,179 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681527331158.a29fa882a9e3f50c391e3aa08c5c5e72. 2023-04-15 02:55:48,179 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-15 02:55:48,179 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-15 02:55:48,179 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:48,179 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-15 02:55:48,179 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34681,1681527209133' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-15 02:55:48,179 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:48,179 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:48,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:48,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:55:48,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:55:48,180 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:48,180 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-15 02:55:48,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:55:48,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:55:48,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:55:48,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:48,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:55:48,181 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34681,1681527209133' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-15 02:55:48,181 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-15 02:55:48,181 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1d414f07[Count = 0] remaining members to acquire global barrier 2023-04-15 02:55:48,181 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:48,182 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:48,182 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:48,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:48,182 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-15 02:55:48,182 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-15 02:55:48,182 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:48,182 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-15 02:55:48,182 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34681,1681527209133' in zk 2023-04-15 02:55:48,183 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:48,183 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-15 02:55:48,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:48,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:55:48,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:55:48,183 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-15 02:55:48,183 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-15 02:55:48,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:55:48,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:55:48,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:55:48,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:48,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:55:48,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:55:48,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:48,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34681,1681527209133': 2023-04-15 02:55:48,185 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34681,1681527209133' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-15 02:55:48,185 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-15 02:55:48,185 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-15 02:55:48,185 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-15 02:55:48,185 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-15 02:55:48,185 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-15 02:55:48,186 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:48,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:48,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:55:48,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:55:48,186 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:48,186 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-15 02:55:48,186 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:48,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:48,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:55:48,187 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:48,187 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-15 02:55:48,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-15 02:55:48,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:55:48,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:48,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:55:48,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:55:48,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:48,187 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:48,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:55:48,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:55:48,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:48,189 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:48,189 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-15 02:55:48,189 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:55:48,189 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-15 02:55:48,189 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-15 02:55:48,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-15 02:55:48,189 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-15 02:55:48,189 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-15 02:55:48,189 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-15 02:55:48,189 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:55:48,189 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-15 02:55:48,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-15 02:55:48,189 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:48,189 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:48,189 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:48,189 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:48,190 DEBUG [Thread-2739] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-15 02:55:48,190 DEBUG [Thread-2739] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-15 02:55:48,290 DEBUG [Thread-2739] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-15 02:55:48,290 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-15 02:55:48,292 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-15 02:55:48,292 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-15 02:55:48,292 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-15 02:55:48,292 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-15 02:55:48,293 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-15 02:55:48,293 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-15 02:55:48,293 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-15 02:55:48,293 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:48,293 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:55:48,294 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:48,294 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-15 02:55:48,294 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-15 02:55:48,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-15 02:55:48,294 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:48,294 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-15 02:55:48,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:55:48,294 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:48,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-15 02:55:48,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:55:48,295 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-15 02:55:48,295 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-15 02:55:48,295 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-15 02:55:48,295 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-15 02:55:48,295 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-15 02:55:48,295 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-15 02:55:48,295 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-15 02:55:48,295 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-15 02:55:48,295 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681527331158.a29fa882a9e3f50c391e3aa08c5c5e72. 2023-04-15 02:55:48,295 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681527331158.a29fa882a9e3f50c391e3aa08c5c5e72. started... 2023-04-15 02:55:48,295 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681527331158.a29fa882a9e3f50c391e3aa08c5c5e72. as already flushing 2023-04-15 02:55:48,295 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681527331158.a29fa882a9e3f50c391e3aa08c5c5e72. 2023-04-15 02:55:48,295 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-15 02:55:48,295 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-15 02:55:48,295 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:48,295 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-15 02:55:48,295 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34681,1681527209133' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-15 02:55:48,296 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:48,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:48,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:55:48,296 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:48,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:55:48,296 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:48,296 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-15 02:55:48,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:55:48,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:55:48,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:55:48,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:48,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:55:48,297 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34681,1681527209133' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-15 02:55:48,297 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-15 02:55:48,297 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@f5d15f2[Count = 0] remaining members to acquire global barrier 2023-04-15 02:55:48,297 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:48,298 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:48,298 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:48,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:48,298 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:48,298 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-15 02:55:48,298 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-15 02:55:48,298 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-15 02:55:48,298 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34681,1681527209133' in zk 2023-04-15 02:55:48,299 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:48,299 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-15 02:55:48,299 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-15 02:55:48,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:48,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:55:48,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:55:48,299 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-15 02:55:48,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:55:48,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:55:48,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:55:48,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:48,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:55:48,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:55:48,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:48,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34681,1681527209133': 2023-04-15 02:55:48,300 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34681,1681527209133' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-15 02:55:48,300 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-15 02:55:48,300 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-15 02:55:48,300 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-15 02:55:48,300 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-15 02:55:48,300 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-15 02:55:48,301 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:48,301 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:48,301 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-15 02:55:48,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:48,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:55:48,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:55:48,301 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:48,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:48,301 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:48,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:55:48,302 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-15 02:55:48,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-15 02:55:48,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:48,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:55:48,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:55:48,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:55:48,302 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:48,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:48,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:55:48,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:55:48,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:48,303 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:48,304 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-15 02:55:48,304 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:55:48,304 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-15 02:55:48,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-15 02:55:48,304 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-15 02:55:48,304 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:55:48,304 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-15 02:55:48,304 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-15 02:55:48,304 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-15 02:55:48,304 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-15 02:55:48,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-15 02:55:48,304 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:48,304 DEBUG [Thread-2739] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-15 02:55:48,304 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:48,304 DEBUG [Thread-2739] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-15 02:55:48,304 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:48,304 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:48,314 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=374 (bloomFilter=true), to=hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/a29fa882a9e3f50c391e3aa08c5c5e72/.tmp/B/4526dc8ea3d145818bce0115e18a3c96 2023-04-15 02:55:48,404 DEBUG [Thread-2739] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-15 02:55:48,405 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-15 02:55:48,406 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-15 02:55:48,407 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-15 02:55:48,407 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-15 02:55:48,407 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-15 02:55:48,407 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-15 02:55:48,407 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-15 02:55:48,407 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-15 02:55:48,407 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:48,407 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:55:48,408 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-15 02:55:48,408 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:48,408 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-15 02:55:48,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-15 02:55:48,408 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:48,408 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-15 02:55:48,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:55:48,408 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:48,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-15 02:55:48,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:55:48,409 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-15 02:55:48,409 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-15 02:55:48,409 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-15 02:55:48,409 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-15 02:55:48,409 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-15 02:55:48,409 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-15 02:55:48,409 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-15 02:55:48,409 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-15 02:55:48,409 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681527331158.a29fa882a9e3f50c391e3aa08c5c5e72. 2023-04-15 02:55:48,409 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681527331158.a29fa882a9e3f50c391e3aa08c5c5e72. started... 2023-04-15 02:55:48,409 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681527331158.a29fa882a9e3f50c391e3aa08c5c5e72. as already flushing 2023-04-15 02:55:48,409 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681527331158.a29fa882a9e3f50c391e3aa08c5c5e72. 2023-04-15 02:55:48,409 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-15 02:55:48,409 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-15 02:55:48,409 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:48,410 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-15 02:55:48,410 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34681,1681527209133' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-15 02:55:48,410 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:48,410 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:48,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:48,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:55:48,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:55:48,411 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:48,411 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-15 02:55:48,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:55:48,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:55:48,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:55:48,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:48,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:55:48,412 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34681,1681527209133' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-15 02:55:48,412 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-15 02:55:48,412 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1f8fd37d[Count = 0] remaining members to acquire global barrier 2023-04-15 02:55:48,412 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:48,412 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:48,412 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:48,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:48,412 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-15 02:55:48,412 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-15 02:55:48,412 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34681,1681527209133' in zk 2023-04-15 02:55:48,413 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:48,413 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-15 02:55:48,413 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-15 02:55:48,413 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-15 02:55:48,413 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:48,413 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-15 02:55:48,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:48,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:55:48,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:55:48,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:55:48,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:55:48,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:55:48,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:48,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:55:48,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:55:48,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:48,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34681,1681527209133': 2023-04-15 02:55:48,415 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34681,1681527209133' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-15 02:55:48,415 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-15 02:55:48,415 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-15 02:55:48,415 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-15 02:55:48,415 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-15 02:55:48,415 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-15 02:55:48,416 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:48,416 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:48,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:48,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:55:48,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:55:48,416 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:48,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:48,416 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-15 02:55:48,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:55:48,416 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-15 02:55:48,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-15 02:55:48,416 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:48,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:55:48,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:48,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:55:48,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:55:48,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:48,417 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:48,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:55:48,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:55:48,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:48,419 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-15 02:55:48,419 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-15 02:55:48,419 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-15 02:55:48,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-15 02:55:48,419 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:48,419 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:55:48,419 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:55:48,419 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-15 02:55:48,419 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-15 02:55:48,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-15 02:55:48,419 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-15 02:55:48,419 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-15 02:55:48,419 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:48,419 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:48,419 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:48,419 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:48,419 DEBUG [Thread-2739] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-15 02:55:48,419 DEBUG [Thread-2739] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-15 02:55:48,519 DEBUG [Thread-2739] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-15 02:55:48,520 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-15 02:55:48,521 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-15 02:55:48,522 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-15 02:55:48,522 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-15 02:55:48,522 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-15 02:55:48,522 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-15 02:55:48,522 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-15 02:55:48,522 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-15 02:55:48,522 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:48,522 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:55:48,523 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-15 02:55:48,523 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:48,523 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-15 02:55:48,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-15 02:55:48,523 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:48,523 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-15 02:55:48,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:55:48,524 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:48,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-15 02:55:48,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:55:48,524 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-15 02:55:48,524 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-15 02:55:48,524 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-15 02:55:48,524 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-15 02:55:48,524 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-15 02:55:48,524 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-15 02:55:48,524 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-15 02:55:48,524 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-15 02:55:48,524 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681527331158.a29fa882a9e3f50c391e3aa08c5c5e72. 2023-04-15 02:55:48,524 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681527331158.a29fa882a9e3f50c391e3aa08c5c5e72. started... 2023-04-15 02:55:48,524 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681527331158.a29fa882a9e3f50c391e3aa08c5c5e72. as already flushing 2023-04-15 02:55:48,524 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681527331158.a29fa882a9e3f50c391e3aa08c5c5e72. 2023-04-15 02:55:48,525 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-15 02:55:48,525 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-15 02:55:48,525 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:48,525 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-15 02:55:48,525 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34681,1681527209133' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-15 02:55:48,525 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:48,525 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:48,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:48,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:55:48,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:55:48,526 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:48,526 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-15 02:55:48,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:55:48,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:55:48,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:55:48,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:48,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:55:48,527 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34681,1681527209133' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-15 02:55:48,527 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-15 02:55:48,527 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6684490d[Count = 0] remaining members to acquire global barrier 2023-04-15 02:55:48,527 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:48,527 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:48,527 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:48,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:48,527 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:48,527 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-15 02:55:48,527 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-15 02:55:48,527 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-15 02:55:48,527 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34681,1681527209133' in zk 2023-04-15 02:55:48,528 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:48,528 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-15 02:55:48,528 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-15 02:55:48,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:48,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:55:48,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:55:48,528 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-15 02:55:48,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:55:48,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:55:48,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:55:48,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:48,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:55:48,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:55:48,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:48,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34681,1681527209133': 2023-04-15 02:55:48,531 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34681,1681527209133' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-15 02:55:48,531 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-15 02:55:48,531 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-15 02:55:48,531 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-15 02:55:48,531 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-15 02:55:48,531 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-15 02:55:48,532 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:48,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:48,532 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:48,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:55:48,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:55:48,532 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:48,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:48,532 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-15 02:55:48,532 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-15 02:55:48,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:55:48,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-15 02:55:48,532 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:48,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:48,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:55:48,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:55:48,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:55:48,533 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:48,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:48,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:55:48,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:55:48,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:48,536 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-15 02:55:48,536 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-15 02:55:48,536 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-15 02:55:48,536 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-15 02:55:48,536 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:48,536 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-15 02:55:48,536 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-15 02:55:48,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-15 02:55:48,536 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-15 02:55:48,536 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:55:48,537 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:55:48,537 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:48,537 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:48,537 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:48,537 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:48,537 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-15 02:55:48,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-15 02:55:48,537 DEBUG [Thread-2739] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-15 02:55:48,537 DEBUG [Thread-2739] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-15 02:55:48,637 DEBUG [Thread-2739] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-15 02:55:48,638 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-15 02:55:48,639 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-15 02:55:48,640 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-15 02:55:48,640 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-15 02:55:48,640 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-15 02:55:48,640 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-15 02:55:48,640 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-15 02:55:48,640 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-15 02:55:48,641 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:48,641 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:55:48,641 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-15 02:55:48,641 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:48,641 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-15 02:55:48,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-15 02:55:48,641 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:48,641 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-15 02:55:48,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:55:48,642 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:48,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-15 02:55:48,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:55:48,642 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-15 02:55:48,643 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-15 02:55:48,643 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-15 02:55:48,643 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-15 02:55:48,643 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-15 02:55:48,643 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-15 02:55:48,643 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-15 02:55:48,643 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-15 02:55:48,643 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681527331158.a29fa882a9e3f50c391e3aa08c5c5e72. 2023-04-15 02:55:48,643 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681527331158.a29fa882a9e3f50c391e3aa08c5c5e72. started... 2023-04-15 02:55:48,643 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681527331158.a29fa882a9e3f50c391e3aa08c5c5e72. as already flushing 2023-04-15 02:55:48,643 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681527331158.a29fa882a9e3f50c391e3aa08c5c5e72. 2023-04-15 02:55:48,643 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-15 02:55:48,643 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-15 02:55:48,643 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:48,643 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-15 02:55:48,643 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34681,1681527209133' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-15 02:55:48,644 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:48,644 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:48,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:48,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:55:48,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:55:48,644 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:48,644 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-15 02:55:48,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:55:48,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:55:48,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:55:48,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:48,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:55:48,645 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34681,1681527209133' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-15 02:55:48,645 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-15 02:55:48,645 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@71aa0bad[Count = 0] remaining members to acquire global barrier 2023-04-15 02:55:48,645 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:48,646 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:48,646 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:48,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:48,646 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-15 02:55:48,646 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-15 02:55:48,646 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34681,1681527209133' in zk 2023-04-15 02:55:48,646 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:48,646 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-15 02:55:48,647 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:48,647 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-15 02:55:48,647 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-15 02:55:48,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:48,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:55:48,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:55:48,647 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-15 02:55:48,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:55:48,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:55:48,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:55:48,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:48,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:55:48,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:55:48,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:48,650 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a29fa882a9e3f50c391e3aa08c5c5e72, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:55:48,650 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34681] ipc.CallRunner(144): callId: 193 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:40234 deadline: 1681527408648, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a29fa882a9e3f50c391e3aa08c5c5e72, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:48,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34681,1681527209133': 2023-04-15 02:55:48,650 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34681,1681527209133' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-15 02:55:48,650 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-15 02:55:48,650 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-15 02:55:48,650 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-15 02:55:48,650 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-15 02:55:48,650 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-15 02:55:48,655 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a29fa882a9e3f50c391e3aa08c5c5e72, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:55:48,655 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34681] ipc.CallRunner(144): callId: 197 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:40210 deadline: 1681527408651, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a29fa882a9e3f50c391e3aa08c5c5e72, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:48,655 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a29fa882a9e3f50c391e3aa08c5c5e72, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:55:48,655 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34681] ipc.CallRunner(144): callId: 195 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:40192 deadline: 1681527408651, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a29fa882a9e3f50c391e3aa08c5c5e72, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:48,664 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:48,664 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:48,664 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:48,664 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-15 02:55:48,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:48,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:48,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:55:48,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:55:48,664 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-15 02:55:48,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-15 02:55:48,664 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:48,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:55:48,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:48,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:55:48,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:55:48,666 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:48,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:55:48,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:48,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:55:48,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:55:48,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:48,671 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:48,671 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-15 02:55:48,671 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:55:48,671 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-15 02:55:48,671 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-15 02:55:48,671 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-15 02:55:48,671 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-15 02:55:48,671 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-15 02:55:48,671 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-15 02:55:48,671 DEBUG [Thread-2739] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-15 02:55:48,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-15 02:55:48,671 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:55:48,671 DEBUG [Thread-2739] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-15 02:55:48,671 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:48,671 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:48,671 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:48,671 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-15 02:55:48,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-15 02:55:48,671 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:48,722 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=374 (bloomFilter=true), to=hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/a29fa882a9e3f50c391e3aa08c5c5e72/.tmp/C/a7ff1aaa2c07441988a294114bfbc8ee 2023-04-15 02:55:48,725 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/a29fa882a9e3f50c391e3aa08c5c5e72/.tmp/A/6b6ea55ba4564b2885f935edea24a8df as hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/a29fa882a9e3f50c391e3aa08c5c5e72/A/6b6ea55ba4564b2885f935edea24a8df 2023-04-15 02:55:48,728 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/a29fa882a9e3f50c391e3aa08c5c5e72/A/6b6ea55ba4564b2885f935edea24a8df, entries=200, sequenceid=374, filesize=14.3 K 2023-04-15 02:55:48,729 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/a29fa882a9e3f50c391e3aa08c5c5e72/.tmp/B/4526dc8ea3d145818bce0115e18a3c96 as hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/a29fa882a9e3f50c391e3aa08c5c5e72/B/4526dc8ea3d145818bce0115e18a3c96 2023-04-15 02:55:48,731 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/a29fa882a9e3f50c391e3aa08c5c5e72/B/4526dc8ea3d145818bce0115e18a3c96, entries=150, sequenceid=374, filesize=11.9 K 2023-04-15 02:55:48,732 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/a29fa882a9e3f50c391e3aa08c5c5e72/.tmp/C/a7ff1aaa2c07441988a294114bfbc8ee as hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/a29fa882a9e3f50c391e3aa08c5c5e72/C/a7ff1aaa2c07441988a294114bfbc8ee 2023-04-15 02:55:48,735 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/a29fa882a9e3f50c391e3aa08c5c5e72/C/a7ff1aaa2c07441988a294114bfbc8ee, entries=150, sequenceid=374, filesize=11.9 K 2023-04-15 02:55:48,735 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~60.38 KB/61830, heapSize ~180 KB/184320, currentSize=114.05 KB/116790 for a29fa882a9e3f50c391e3aa08c5c5e72 in 1239ms, sequenceid=374, compaction requested=true 2023-04-15 02:55:48,735 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for a29fa882a9e3f50c391e3aa08c5c5e72: 2023-04-15 02:55:48,735 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-15 02:55:48,736 DEBUG [RS:0;jenkins-hbase20:34681-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-15 02:55:48,736 DEBUG [RS:0;jenkins-hbase20:34681-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-15 02:55:48,736 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-15 02:55:48,736 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-15 02:55:48,736 DEBUG [RS:0;jenkins-hbase20:34681-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 39680 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-15 02:55:48,736 DEBUG [RS:0;jenkins-hbase20:34681-shortCompactions-0] regionserver.HStore(1912): a29fa882a9e3f50c391e3aa08c5c5e72/A is initiating minor compaction (all files) 2023-04-15 02:55:48,737 DEBUG [RS:0;jenkins-hbase20:34681-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37240 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-15 02:55:48,737 INFO [RS:0;jenkins-hbase20:34681-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of a29fa882a9e3f50c391e3aa08c5c5e72/A in TestAcidGuarantees,,1681527331158.a29fa882a9e3f50c391e3aa08c5c5e72. 2023-04-15 02:55:48,737 DEBUG [RS:0;jenkins-hbase20:34681-longCompactions-0] regionserver.HStore(1912): a29fa882a9e3f50c391e3aa08c5c5e72/B is initiating minor compaction (all files) 2023-04-15 02:55:48,737 INFO [RS:0;jenkins-hbase20:34681-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/a29fa882a9e3f50c391e3aa08c5c5e72/A/1ee1d258316345379a04f4b9bd87e5ed, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/a29fa882a9e3f50c391e3aa08c5c5e72/A/e92ab01b17d04e6692817e2ec14e1214, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/a29fa882a9e3f50c391e3aa08c5c5e72/A/6b6ea55ba4564b2885f935edea24a8df] into tmpdir=hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/a29fa882a9e3f50c391e3aa08c5c5e72/.tmp, totalSize=38.8 K 2023-04-15 02:55:48,737 INFO [RS:0;jenkins-hbase20:34681-longCompactions-0] regionserver.HRegion(2259): Starting compaction of a29fa882a9e3f50c391e3aa08c5c5e72/B in TestAcidGuarantees,,1681527331158.a29fa882a9e3f50c391e3aa08c5c5e72. 2023-04-15 02:55:48,737 INFO [RS:0;jenkins-hbase20:34681-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/a29fa882a9e3f50c391e3aa08c5c5e72/B/f6e80b7413c74242b9c7c04ee2582e18, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/a29fa882a9e3f50c391e3aa08c5c5e72/B/6ba49c87f56a4a3cb7b3c3b548430ce1, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/a29fa882a9e3f50c391e3aa08c5c5e72/B/4526dc8ea3d145818bce0115e18a3c96] into tmpdir=hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/a29fa882a9e3f50c391e3aa08c5c5e72/.tmp, totalSize=36.4 K 2023-04-15 02:55:48,737 DEBUG [RS:0;jenkins-hbase20:34681-shortCompactions-0] compactions.Compactor(207): Compacting 1ee1d258316345379a04f4b9bd87e5ed, keycount=150, bloomtype=ROW, size=12.6 K, encoding=NONE, compression=NONE, seqNum=323, earliestPutTs=1681527345671 2023-04-15 02:55:48,737 DEBUG [RS:0;jenkins-hbase20:34681-longCompactions-0] compactions.Compactor(207): Compacting f6e80b7413c74242b9c7c04ee2582e18, keycount=150, bloomtype=ROW, size=12.6 K, encoding=NONE, compression=NONE, seqNum=323, earliestPutTs=1681527345671 2023-04-15 02:55:48,737 DEBUG [RS:0;jenkins-hbase20:34681-shortCompactions-0] compactions.Compactor(207): Compacting e92ab01b17d04e6692817e2ec14e1214, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=361, earliestPutTs=1681527345726 2023-04-15 02:55:48,738 DEBUG [RS:0;jenkins-hbase20:34681-longCompactions-0] compactions.Compactor(207): Compacting 6ba49c87f56a4a3cb7b3c3b548430ce1, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=361, earliestPutTs=1681527345726 2023-04-15 02:55:48,738 DEBUG [RS:0;jenkins-hbase20:34681-shortCompactions-0] compactions.Compactor(207): Compacting 6b6ea55ba4564b2885f935edea24a8df, keycount=200, bloomtype=ROW, size=14.3 K, encoding=NONE, compression=NONE, seqNum=374, earliestPutTs=1681527346878 2023-04-15 02:55:48,738 DEBUG [RS:0;jenkins-hbase20:34681-longCompactions-0] compactions.Compactor(207): Compacting 4526dc8ea3d145818bce0115e18a3c96, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=374, earliestPutTs=1681527346880 2023-04-15 02:55:48,742 INFO [RS:0;jenkins-hbase20:34681-longCompactions-0] throttle.PressureAwareThroughputController(145): a29fa882a9e3f50c391e3aa08c5c5e72#B#compaction#413 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-15 02:55:48,743 INFO [RS:0;jenkins-hbase20:34681-shortCompactions-0] throttle.PressureAwareThroughputController(145): a29fa882a9e3f50c391e3aa08c5c5e72#A#compaction#414 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-15 02:55:48,760 DEBUG [RS:0;jenkins-hbase20:34681-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/a29fa882a9e3f50c391e3aa08c5c5e72/.tmp/A/8b168677d78840c7bc3dba0bfc7fa4cc as hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/a29fa882a9e3f50c391e3aa08c5c5e72/A/8b168677d78840c7bc3dba0bfc7fa4cc 2023-04-15 02:55:48,763 INFO [RS:0;jenkins-hbase20:34681-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in a29fa882a9e3f50c391e3aa08c5c5e72/A of a29fa882a9e3f50c391e3aa08c5c5e72 into 8b168677d78840c7bc3dba0bfc7fa4cc(size=12.7 K), total size for store is 12.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-15 02:55:48,763 DEBUG [RS:0;jenkins-hbase20:34681-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for a29fa882a9e3f50c391e3aa08c5c5e72: 2023-04-15 02:55:48,763 INFO [RS:0;jenkins-hbase20:34681-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681527331158.a29fa882a9e3f50c391e3aa08c5c5e72., storeName=a29fa882a9e3f50c391e3aa08c5c5e72/A, priority=13, startTime=1681527348735; duration=0sec 2023-04-15 02:55:48,763 DEBUG [RS:0;jenkins-hbase20:34681-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-15 02:55:48,763 DEBUG [RS:0;jenkins-hbase20:34681-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-15 02:55:48,764 DEBUG [RS:0;jenkins-hbase20:34681-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37240 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-15 02:55:48,764 DEBUG [RS:0;jenkins-hbase20:34681-shortCompactions-0] regionserver.HStore(1912): a29fa882a9e3f50c391e3aa08c5c5e72/C is initiating minor compaction (all files) 2023-04-15 02:55:48,764 INFO [RS:0;jenkins-hbase20:34681-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of a29fa882a9e3f50c391e3aa08c5c5e72/C in TestAcidGuarantees,,1681527331158.a29fa882a9e3f50c391e3aa08c5c5e72. 2023-04-15 02:55:48,764 INFO [RS:0;jenkins-hbase20:34681-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/a29fa882a9e3f50c391e3aa08c5c5e72/C/0bb7d65eea50491994aff94d3a8396a5, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/a29fa882a9e3f50c391e3aa08c5c5e72/C/49455a4d121741a59906287bbd6150b2, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/a29fa882a9e3f50c391e3aa08c5c5e72/C/a7ff1aaa2c07441988a294114bfbc8ee] into tmpdir=hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/a29fa882a9e3f50c391e3aa08c5c5e72/.tmp, totalSize=36.4 K 2023-04-15 02:55:48,765 DEBUG [RS:0;jenkins-hbase20:34681-shortCompactions-0] compactions.Compactor(207): Compacting 0bb7d65eea50491994aff94d3a8396a5, keycount=150, bloomtype=ROW, size=12.6 K, encoding=NONE, compression=NONE, seqNum=323, earliestPutTs=1681527345671 2023-04-15 02:55:48,765 DEBUG [RS:0;jenkins-hbase20:34681-shortCompactions-0] compactions.Compactor(207): Compacting 49455a4d121741a59906287bbd6150b2, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=361, earliestPutTs=1681527345726 2023-04-15 02:55:48,765 DEBUG [RS:0;jenkins-hbase20:34681-shortCompactions-0] compactions.Compactor(207): Compacting a7ff1aaa2c07441988a294114bfbc8ee, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=374, earliestPutTs=1681527346880 2023-04-15 02:55:48,769 INFO [RS:0;jenkins-hbase20:34681-shortCompactions-0] throttle.PressureAwareThroughputController(145): a29fa882a9e3f50c391e3aa08c5c5e72#C#compaction#415 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-15 02:55:48,771 DEBUG [Thread-2739] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-15 02:55:48,772 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-15 02:55:48,773 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-15 02:55:48,774 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-15 02:55:48,774 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-15 02:55:48,774 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-15 02:55:48,774 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-15 02:55:48,774 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-15 02:55:48,774 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-15 02:55:48,774 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:48,774 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:55:48,775 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-15 02:55:48,775 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-15 02:55:48,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-15 02:55:48,775 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:48,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:55:48,776 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:48,776 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:48,776 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-15 02:55:48,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-15 02:55:48,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:55:48,777 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-15 02:55:48,778 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-15 02:55:48,778 DEBUG [RS:0;jenkins-hbase20:34681-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/a29fa882a9e3f50c391e3aa08c5c5e72/.tmp/C/35cbbdf3250541bb914fe4d9c46f472c as hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/a29fa882a9e3f50c391e3aa08c5c5e72/C/35cbbdf3250541bb914fe4d9c46f472c 2023-04-15 02:55:48,778 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-15 02:55:48,778 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-15 02:55:48,778 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-15 02:55:48,778 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-15 02:55:48,779 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-15 02:55:48,779 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-15 02:55:48,779 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681527331158.a29fa882a9e3f50c391e3aa08c5c5e72. 2023-04-15 02:55:48,779 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681527331158.a29fa882a9e3f50c391e3aa08c5c5e72. started... 2023-04-15 02:55:48,779 INFO [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing a29fa882a9e3f50c391e3aa08c5c5e72 3/3 column families, dataSize=114.05 KB heapSize=339.42 KB 2023-04-15 02:55:48,779 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a29fa882a9e3f50c391e3aa08c5c5e72, store=A 2023-04-15 02:55:48,779 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-15 02:55:48,779 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a29fa882a9e3f50c391e3aa08c5c5e72, store=B 2023-04-15 02:55:48,779 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-15 02:55:48,781 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a29fa882a9e3f50c391e3aa08c5c5e72, store=C 2023-04-15 02:55:48,781 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-15 02:55:48,781 INFO [RS:0;jenkins-hbase20:34681-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in a29fa882a9e3f50c391e3aa08c5c5e72/C of a29fa882a9e3f50c391e3aa08c5c5e72 into 35cbbdf3250541bb914fe4d9c46f472c(size=12.7 K), total size for store is 12.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-15 02:55:48,781 DEBUG [RS:0;jenkins-hbase20:34681-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for a29fa882a9e3f50c391e3aa08c5c5e72: 2023-04-15 02:55:48,782 INFO [RS:0;jenkins-hbase20:34681-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681527331158.a29fa882a9e3f50c391e3aa08c5c5e72., storeName=a29fa882a9e3f50c391e3aa08c5c5e72/C, priority=13, startTime=1681527348736; duration=0sec 2023-04-15 02:55:48,782 DEBUG [RS:0;jenkins-hbase20:34681-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-15 02:55:48,787 INFO [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=395 (bloomFilter=true), to=hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/a29fa882a9e3f50c391e3aa08c5c5e72/.tmp/A/90e4b2984b0b4171b9e59884e699b38b 2023-04-15 02:55:48,803 INFO [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=395 (bloomFilter=true), to=hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/a29fa882a9e3f50c391e3aa08c5c5e72/.tmp/B/513f80e4f4a141dc97f9f4db67f57ba6 2023-04-15 02:55:49,159 DEBUG [RS:0;jenkins-hbase20:34681-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/a29fa882a9e3f50c391e3aa08c5c5e72/.tmp/B/558068e1a6fd477385b4c9b7a02a5be1 as hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/a29fa882a9e3f50c391e3aa08c5c5e72/B/558068e1a6fd477385b4c9b7a02a5be1 2023-04-15 02:55:49,162 INFO [RS:0;jenkins-hbase20:34681-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in a29fa882a9e3f50c391e3aa08c5c5e72/B of a29fa882a9e3f50c391e3aa08c5c5e72 into 558068e1a6fd477385b4c9b7a02a5be1(size=12.7 K), total size for store is 12.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-15 02:55:49,163 DEBUG [RS:0;jenkins-hbase20:34681-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for a29fa882a9e3f50c391e3aa08c5c5e72: 2023-04-15 02:55:49,163 INFO [RS:0;jenkins-hbase20:34681-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681527331158.a29fa882a9e3f50c391e3aa08c5c5e72., storeName=a29fa882a9e3f50c391e3aa08c5c5e72/B, priority=13, startTime=1681527348736; duration=0sec 2023-04-15 02:55:49,163 DEBUG [RS:0;jenkins-hbase20:34681-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-15 02:55:49,211 INFO [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=395 (bloomFilter=true), to=hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/a29fa882a9e3f50c391e3aa08c5c5e72/.tmp/C/35d587b695f74c159d812876bf3420cc 2023-04-15 02:55:49,215 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/a29fa882a9e3f50c391e3aa08c5c5e72/.tmp/A/90e4b2984b0b4171b9e59884e699b38b as hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/a29fa882a9e3f50c391e3aa08c5c5e72/A/90e4b2984b0b4171b9e59884e699b38b 2023-04-15 02:55:49,217 INFO [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/a29fa882a9e3f50c391e3aa08c5c5e72/A/90e4b2984b0b4171b9e59884e699b38b, entries=150, sequenceid=395, filesize=11.9 K 2023-04-15 02:55:49,218 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/a29fa882a9e3f50c391e3aa08c5c5e72/.tmp/B/513f80e4f4a141dc97f9f4db67f57ba6 as hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/a29fa882a9e3f50c391e3aa08c5c5e72/B/513f80e4f4a141dc97f9f4db67f57ba6 2023-04-15 02:55:49,220 INFO [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/a29fa882a9e3f50c391e3aa08c5c5e72/B/513f80e4f4a141dc97f9f4db67f57ba6, entries=150, sequenceid=395, filesize=11.9 K 2023-04-15 02:55:49,221 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/a29fa882a9e3f50c391e3aa08c5c5e72/.tmp/C/35d587b695f74c159d812876bf3420cc as hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/a29fa882a9e3f50c391e3aa08c5c5e72/C/35d587b695f74c159d812876bf3420cc 2023-04-15 02:55:49,223 INFO [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/a29fa882a9e3f50c391e3aa08c5c5e72/C/35d587b695f74c159d812876bf3420cc, entries=150, sequenceid=395, filesize=11.9 K 2023-04-15 02:55:49,224 INFO [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~114.05 KB/116790, heapSize ~339.38 KB/347520, currentSize=0 B/0 for a29fa882a9e3f50c391e3aa08c5c5e72 in 445ms, sequenceid=395, compaction requested=false 2023-04-15 02:55:49,224 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for a29fa882a9e3f50c391e3aa08c5c5e72: 2023-04-15 02:55:49,224 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681527331158.a29fa882a9e3f50c391e3aa08c5c5e72. 2023-04-15 02:55:49,224 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-15 02:55:49,224 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-15 02:55:49,224 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:49,224 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-15 02:55:49,224 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34681,1681527209133' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-15 02:55:49,227 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:49,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:49,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:55:49,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:55:49,227 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:49,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:55:49,227 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:49,227 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-15 02:55:49,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:55:49,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:55:49,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:49,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:55:49,228 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34681,1681527209133' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-15 02:55:49,228 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@163f72ef[Count = 0] remaining members to acquire global barrier 2023-04-15 02:55:49,228 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-15 02:55:49,228 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:49,229 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:49,229 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:49,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:49,229 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-15 02:55:49,229 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-15 02:55:49,229 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:49,229 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-15 02:55:49,229 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34681,1681527209133' in zk 2023-04-15 02:55:49,229 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:49,230 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-15 02:55:49,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:49,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:55:49,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:55:49,230 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-15 02:55:49,230 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-15 02:55:49,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:55:49,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:55:49,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:55:49,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:49,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:55:49,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:55:49,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:49,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34681,1681527209133': 2023-04-15 02:55:49,231 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34681,1681527209133' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-15 02:55:49,231 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-15 02:55:49,231 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-15 02:55:49,231 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-15 02:55:49,232 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-15 02:55:49,232 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-15 02:55:49,232 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:49,232 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:49,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:49,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:55:49,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:55:49,232 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:49,232 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-15 02:55:49,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:49,234 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-15 02:55:49,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-15 02:55:49,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:55:49,234 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:49,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:49,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:55:49,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:55:49,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:55:49,235 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:49,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:49,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:55:49,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:55:49,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:49,236 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:49,236 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:55:49,236 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:55:49,236 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-15 02:55:49,236 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-15 02:55:49,236 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-15 02:55:49,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-15 02:55:49,236 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-15 02:55:49,236 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-15 02:55:49,236 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:49,236 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:49,237 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:49,237 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:49,237 DEBUG [Thread-2739] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-15 02:55:49,236 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-15 02:55:49,236 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-15 02:55:49,237 DEBUG [Thread-2739] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-15 02:55:49,236 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-15 02:55:49,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-15 02:55:49,337 DEBUG [Thread-2739] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-15 02:55:49,337 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-15 02:55:49,338 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-15 02:55:49,339 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-15 02:55:49,339 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-15 02:55:49,339 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-15 02:55:49,339 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-15 02:55:49,339 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-15 02:55:49,339 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-15 02:55:49,340 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:49,340 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:55:49,340 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:49,340 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-15 02:55:49,341 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-15 02:55:49,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-15 02:55:49,341 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:49,341 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-15 02:55:49,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:55:49,341 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:49,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-15 02:55:49,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:55:49,341 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-15 02:55:49,342 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-15 02:55:49,342 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-15 02:55:49,342 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-15 02:55:49,342 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-15 02:55:49,342 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-15 02:55:49,343 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-15 02:55:49,343 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-15 02:55:49,343 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681527331158.a29fa882a9e3f50c391e3aa08c5c5e72. 2023-04-15 02:55:49,343 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681527331158.a29fa882a9e3f50c391e3aa08c5c5e72. started... 2023-04-15 02:55:49,343 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for a29fa882a9e3f50c391e3aa08c5c5e72: 2023-04-15 02:55:49,343 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681527331158.a29fa882a9e3f50c391e3aa08c5c5e72. 2023-04-15 02:55:49,344 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-15 02:55:49,344 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-15 02:55:49,344 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:49,345 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-15 02:55:49,345 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34681,1681527209133' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-15 02:55:49,345 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:49,345 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:49,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:49,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:55:49,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:55:49,346 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:49,346 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-15 02:55:49,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:55:49,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:55:49,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:55:49,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:49,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:55:49,347 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34681,1681527209133' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-15 02:55:49,347 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@77ca5f6e[Count = 0] remaining members to acquire global barrier 2023-04-15 02:55:49,347 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-15 02:55:49,347 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:49,348 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:49,348 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:49,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:49,348 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-15 02:55:49,348 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:49,348 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-15 02:55:49,348 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-15 02:55:49,348 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34681,1681527209133' in zk 2023-04-15 02:55:49,349 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-15 02:55:49,349 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-15 02:55:49,349 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:49,349 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-15 02:55:49,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:49,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:55:49,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:55:49,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:55:49,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:55:49,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:55:49,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:49,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:55:49,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:55:49,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:49,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34681,1681527209133': 2023-04-15 02:55:49,350 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34681,1681527209133' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-15 02:55:49,350 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-15 02:55:49,350 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-15 02:55:49,350 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-15 02:55:49,351 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-15 02:55:49,351 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-15 02:55:49,351 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:49,351 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:49,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:49,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:55:49,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:55:49,351 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:49,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:49,351 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-15 02:55:49,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:55:49,352 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:49,352 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-15 02:55:49,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-15 02:55:49,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:55:49,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:49,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:55:49,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:55:49,353 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:49,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:49,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:55:49,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:55:49,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:49,355 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:49,355 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:55:49,355 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:55:49,355 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:49,355 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:49,355 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-15 02:55:49,355 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-15 02:55:49,355 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:49,355 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-15 02:55:49,355 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-15 02:55:49,355 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:49,355 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-15 02:55:49,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-15 02:55:49,355 DEBUG [Thread-2739] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-15 02:55:49,355 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-15 02:55:49,355 DEBUG [Thread-2739] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-15 02:55:49,356 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-15 02:55:49,356 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-15 02:55:49,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-15 02:55:49,456 DEBUG [Thread-2739] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-15 02:55:49,456 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-15 02:55:49,457 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-15 02:55:49,458 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-15 02:55:49,458 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-15 02:55:49,458 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-15 02:55:49,458 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-15 02:55:49,458 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-15 02:55:49,458 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-15 02:55:49,459 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:49,459 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:55:49,459 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-15 02:55:49,459 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:49,459 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-15 02:55:49,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-15 02:55:49,460 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:49,460 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-15 02:55:49,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:55:49,460 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:49,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-15 02:55:49,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:55:49,460 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-15 02:55:49,460 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-15 02:55:49,460 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-15 02:55:49,460 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-15 02:55:49,460 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-15 02:55:49,461 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-15 02:55:49,461 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-15 02:55:49,461 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681527331158.a29fa882a9e3f50c391e3aa08c5c5e72. 2023-04-15 02:55:49,461 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681527331158.a29fa882a9e3f50c391e3aa08c5c5e72. started... 2023-04-15 02:55:49,461 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for a29fa882a9e3f50c391e3aa08c5c5e72: 2023-04-15 02:55:49,462 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681527331158.a29fa882a9e3f50c391e3aa08c5c5e72. 2023-04-15 02:55:49,462 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-15 02:55:49,462 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-15 02:55:49,462 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:49,462 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-15 02:55:49,462 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34681,1681527209133' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-15 02:55:49,463 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:49,463 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:49,463 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:49,463 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-15 02:55:49,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:49,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:55:49,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:55:49,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:55:49,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:55:49,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:55:49,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:49,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:55:49,467 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34681,1681527209133' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-15 02:55:49,467 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-15 02:55:49,467 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3342acbb[Count = 0] remaining members to acquire global barrier 2023-04-15 02:55:49,467 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:49,468 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:49,468 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:49,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:49,468 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-15 02:55:49,468 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-15 02:55:49,468 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34681,1681527209133' in zk 2023-04-15 02:55:49,468 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:49,468 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-15 02:55:49,469 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:49,469 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-15 02:55:49,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:49,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:55:49,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:55:49,469 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-15 02:55:49,469 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-15 02:55:49,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:55:49,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:55:49,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:55:49,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:49,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:55:49,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:55:49,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:49,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34681,1681527209133': 2023-04-15 02:55:49,471 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34681,1681527209133' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-15 02:55:49,471 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-15 02:55:49,471 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-15 02:55:49,471 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-15 02:55:49,471 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-15 02:55:49,471 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-15 02:55:49,472 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:49,472 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:49,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:49,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:55:49,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:55:49,472 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:49,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:49,472 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-15 02:55:49,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:55:49,472 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-15 02:55:49,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-15 02:55:49,472 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:49,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:49,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:55:49,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:55:49,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:55:49,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:49,473 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:49,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:55:49,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:55:49,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:49,475 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-15 02:55:49,475 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-15 02:55:49,475 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-15 02:55:49,476 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:49,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-15 02:55:49,476 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:55:49,476 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-15 02:55:49,476 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-15 02:55:49,476 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-15 02:55:49,476 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-15 02:55:49,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-15 02:55:49,476 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:55:49,476 DEBUG [Thread-2739] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-15 02:55:49,476 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:49,476 DEBUG [Thread-2739] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-15 02:55:49,476 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:49,476 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:49,476 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:49,576 DEBUG [Thread-2739] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-15 02:55:49,577 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-15 02:55:49,578 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-15 02:55:49,579 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-15 02:55:49,579 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-15 02:55:49,579 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-15 02:55:49,579 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-15 02:55:49,579 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-15 02:55:49,579 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-15 02:55:49,579 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:49,579 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:55:49,581 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-15 02:55:49,581 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:49,581 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-15 02:55:49,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-15 02:55:49,582 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:49,582 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-15 02:55:49,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:55:49,582 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:49,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-15 02:55:49,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:55:49,583 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-15 02:55:49,583 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-15 02:55:49,583 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-15 02:55:49,583 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-15 02:55:49,583 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-15 02:55:49,583 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-15 02:55:49,583 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-15 02:55:49,583 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-15 02:55:49,583 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681527331158.a29fa882a9e3f50c391e3aa08c5c5e72. 2023-04-15 02:55:49,583 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681527331158.a29fa882a9e3f50c391e3aa08c5c5e72. started... 2023-04-15 02:55:49,584 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for a29fa882a9e3f50c391e3aa08c5c5e72: 2023-04-15 02:55:49,584 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681527331158.a29fa882a9e3f50c391e3aa08c5c5e72. 2023-04-15 02:55:49,584 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-15 02:55:49,584 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-15 02:55:49,584 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:49,584 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-15 02:55:49,584 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34681,1681527209133' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-15 02:55:49,585 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:49,585 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:49,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:49,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:55:49,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:55:49,585 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:49,586 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-15 02:55:49,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:55:49,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:55:49,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:55:49,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:49,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:55:49,587 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34681,1681527209133' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-15 02:55:49,587 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-15 02:55:49,587 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@12942e1d[Count = 0] remaining members to acquire global barrier 2023-04-15 02:55:49,587 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:49,588 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:49,588 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:49,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:49,588 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-15 02:55:49,588 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-15 02:55:49,588 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34681,1681527209133' in zk 2023-04-15 02:55:49,588 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:49,588 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-15 02:55:49,589 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:49,589 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-15 02:55:49,589 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-15 02:55:49,589 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-15 02:55:49,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:49,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:55:49,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:55:49,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:55:49,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:55:49,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:55:49,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:49,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:55:49,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:55:49,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:49,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34681,1681527209133': 2023-04-15 02:55:49,591 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34681,1681527209133' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-15 02:55:49,591 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-15 02:55:49,591 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-15 02:55:49,591 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-15 02:55:49,591 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-15 02:55:49,591 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-15 02:55:49,593 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:49,593 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:49,593 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-15 02:55:49,593 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:49,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:49,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:49,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:55:49,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:55:49,593 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-15 02:55:49,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-15 02:55:49,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:55:49,593 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:49,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:49,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:55:49,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:55:49,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:55:49,595 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:49,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:49,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:55:49,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:55:49,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:49,596 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-15 02:55:49,596 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-15 02:55:49,596 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:49,596 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:55:49,596 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:55:49,596 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:49,596 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:49,596 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:49,596 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:49,596 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-15 02:55:49,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-15 02:55:49,596 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-15 02:55:49,596 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-15 02:55:49,597 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-15 02:55:49,596 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-15 02:55:49,597 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-15 02:55:49,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-15 02:55:49,597 DEBUG [Thread-2739] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-15 02:55:49,597 DEBUG [Thread-2739] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-15 02:55:49,667 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34681] regionserver.HRegion(9158): Flush requested on a29fa882a9e3f50c391e3aa08c5c5e72 2023-04-15 02:55:49,667 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing a29fa882a9e3f50c391e3aa08c5c5e72 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-15 02:55:49,667 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a29fa882a9e3f50c391e3aa08c5c5e72, store=A 2023-04-15 02:55:49,667 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-15 02:55:49,667 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a29fa882a9e3f50c391e3aa08c5c5e72, store=B 2023-04-15 02:55:49,667 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-15 02:55:49,667 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a29fa882a9e3f50c391e3aa08c5c5e72, store=C 2023-04-15 02:55:49,667 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-15 02:55:49,694 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1681527349678, max timestamp=1681527349693Num uniques 859; 2023-04-15 02:55:49,695 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-7] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-15 02:55:49,695 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1681527349678, max timestamp=1681527349693Num uniques 859; 2023-04-15 02:55:49,695 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1681527349678, max timestamp=1681527349693Num uniques 859; 2023-04-15 02:55:49,695 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-1] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-15 02:55:49,695 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-15 02:55:49,697 DEBUG [Thread-2739] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-15 02:55:49,698 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-15 02:55:49,699 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-15 02:55:49,700 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-15 02:55:49,700 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-15 02:55:49,700 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-15 02:55:49,700 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-15 02:55:49,700 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-15 02:55:49,700 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-15 02:55:49,700 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:49,700 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:55:49,701 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:49,701 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-15 02:55:49,701 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-15 02:55:49,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-15 02:55:49,701 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:49,701 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-15 02:55:49,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:55:49,701 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:49,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-15 02:55:49,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:55:49,702 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-15 02:55:49,702 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-15 02:55:49,702 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-15 02:55:49,702 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-15 02:55:49,702 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-15 02:55:49,702 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-15 02:55:49,702 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-15 02:55:49,702 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-15 02:55:49,702 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681527331158.a29fa882a9e3f50c391e3aa08c5c5e72. 2023-04-15 02:55:49,702 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681527331158.a29fa882a9e3f50c391e3aa08c5c5e72. started... 2023-04-15 02:55:49,702 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681527331158.a29fa882a9e3f50c391e3aa08c5c5e72. as already flushing 2023-04-15 02:55:49,702 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681527331158.a29fa882a9e3f50c391e3aa08c5c5e72. 2023-04-15 02:55:49,702 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-15 02:55:49,702 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-15 02:55:49,702 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:49,702 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-15 02:55:49,703 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34681,1681527209133' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-15 02:55:49,703 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:49,703 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:49,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:49,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:55:49,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:55:49,704 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:49,704 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-15 02:55:49,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:55:49,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:55:49,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:55:49,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:49,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:55:49,704 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34681,1681527209133' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-15 02:55:49,705 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-15 02:55:49,705 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7fd19eaa[Count = 0] remaining members to acquire global barrier 2023-04-15 02:55:49,705 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:49,705 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:49,705 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:49,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:49,705 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-15 02:55:49,705 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-15 02:55:49,705 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:49,705 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-15 02:55:49,705 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34681,1681527209133' in zk 2023-04-15 02:55:49,706 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:49,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:49,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:55:49,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:55:49,706 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-15 02:55:49,706 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-15 02:55:49,706 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-15 02:55:49,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:55:49,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:55:49,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:55:49,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:49,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:55:49,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:55:49,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:49,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34681,1681527209133': 2023-04-15 02:55:49,709 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34681,1681527209133' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-15 02:55:49,709 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-15 02:55:49,709 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-15 02:55:49,709 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-15 02:55:49,709 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-15 02:55:49,709 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-15 02:55:49,710 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:49,710 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:49,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:49,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:55:49,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:55:49,710 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:49,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:49,710 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-15 02:55:49,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:55:49,711 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:49,711 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-15 02:55:49,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-15 02:55:49,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:55:49,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:49,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:55:49,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:55:49,711 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:49,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:49,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:55:49,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:55:49,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:49,713 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:49,713 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:55:49,713 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-15 02:55:49,713 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:55:49,713 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-15 02:55:49,713 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:49,713 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:49,713 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-15 02:55:49,713 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-15 02:55:49,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-15 02:55:49,713 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:49,713 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:49,713 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-15 02:55:49,713 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-15 02:55:49,713 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-15 02:55:49,713 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-15 02:55:49,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-15 02:55:49,713 DEBUG [Thread-2739] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-15 02:55:49,713 DEBUG [Thread-2739] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-15 02:55:49,726 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a29fa882a9e3f50c391e3aa08c5c5e72, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:55:49,726 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34681] ipc.CallRunner(144): callId: 210 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:40210 deadline: 1681527409721, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a29fa882a9e3f50c391e3aa08c5c5e72, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:49,726 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a29fa882a9e3f50c391e3aa08c5c5e72, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:55:49,727 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34681] ipc.CallRunner(144): callId: 212 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:40234 deadline: 1681527409721, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a29fa882a9e3f50c391e3aa08c5c5e72, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:49,728 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a29fa882a9e3f50c391e3aa08c5c5e72, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:55:49,728 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34681] ipc.CallRunner(144): callId: 209 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:40192 deadline: 1681527409722, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a29fa882a9e3f50c391e3aa08c5c5e72, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:49,813 DEBUG [Thread-2739] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-15 02:55:49,814 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-15 02:55:49,815 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-15 02:55:49,816 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-15 02:55:49,816 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-15 02:55:49,816 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-15 02:55:49,816 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-15 02:55:49,816 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-15 02:55:49,816 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-15 02:55:49,817 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:49,817 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:55:49,817 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-15 02:55:49,817 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:49,817 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-15 02:55:49,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-15 02:55:49,818 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:49,818 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-15 02:55:49,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:55:49,818 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:49,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-15 02:55:49,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:55:49,818 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-15 02:55:49,818 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-15 02:55:49,818 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-15 02:55:49,818 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-15 02:55:49,818 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-15 02:55:49,818 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-15 02:55:49,818 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-15 02:55:49,819 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-15 02:55:49,819 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681527331158.a29fa882a9e3f50c391e3aa08c5c5e72. 2023-04-15 02:55:49,819 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681527331158.a29fa882a9e3f50c391e3aa08c5c5e72. started... 2023-04-15 02:55:49,819 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681527331158.a29fa882a9e3f50c391e3aa08c5c5e72. as already flushing 2023-04-15 02:55:49,819 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681527331158.a29fa882a9e3f50c391e3aa08c5c5e72. 2023-04-15 02:55:49,819 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-15 02:55:49,819 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-15 02:55:49,819 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:49,819 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-15 02:55:49,819 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34681,1681527209133' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-15 02:55:49,820 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:49,820 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:49,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:49,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:55:49,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:55:49,820 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:49,820 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-15 02:55:49,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:55:49,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:55:49,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:55:49,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:49,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:55:49,822 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34681,1681527209133' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-15 02:55:49,822 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5a458f11[Count = 0] remaining members to acquire global barrier 2023-04-15 02:55:49,822 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-15 02:55:49,822 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:49,823 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:49,823 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:49,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:49,823 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-15 02:55:49,823 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-15 02:55:49,823 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34681,1681527209133' in zk 2023-04-15 02:55:49,823 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:49,823 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-15 02:55:49,824 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-15 02:55:49,824 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-15 02:55:49,824 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:49,824 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-15 02:55:49,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:49,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:55:49,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:55:49,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:55:49,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:55:49,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:55:49,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:49,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:55:49,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:55:49,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:49,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34681,1681527209133': 2023-04-15 02:55:49,825 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34681,1681527209133' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-15 02:55:49,825 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-15 02:55:49,826 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-15 02:55:49,826 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-15 02:55:49,826 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-15 02:55:49,826 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-15 02:55:49,826 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:49,826 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:49,826 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:49,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:49,826 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-15 02:55:49,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:49,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:55:49,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:55:49,827 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-15 02:55:49,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-15 02:55:49,827 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:49,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:49,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:55:49,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:55:49,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:55:49,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:55:49,829 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:49,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:49,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:55:49,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:55:49,829 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a29fa882a9e3f50c391e3aa08c5c5e72, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:55:49,829 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34681] ipc.CallRunner(144): callId: 214 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:40234 deadline: 1681527409827, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a29fa882a9e3f50c391e3aa08c5c5e72, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:49,830 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:49,830 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:55:49,831 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:55:49,831 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:49,831 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:49,831 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:49,831 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-15 02:55:49,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:49,831 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-15 02:55:49,831 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-15 02:55:49,831 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:49,831 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-15 02:55:49,831 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-15 02:55:49,831 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-15 02:55:49,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-15 02:55:49,831 DEBUG [Thread-2739] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-15 02:55:49,831 DEBUG [Thread-2739] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-15 02:55:49,831 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-15 02:55:49,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-15 02:55:49,831 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-15 02:55:49,833 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a29fa882a9e3f50c391e3aa08c5c5e72, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:55:49,833 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34681] ipc.CallRunner(144): callId: 212 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:40210 deadline: 1681527409828, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a29fa882a9e3f50c391e3aa08c5c5e72, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:49,833 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a29fa882a9e3f50c391e3aa08c5c5e72, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:55:49,833 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34681] ipc.CallRunner(144): callId: 211 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:40192 deadline: 1681527409829, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a29fa882a9e3f50c391e3aa08c5c5e72, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:49,931 DEBUG [Thread-2739] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-15 02:55:49,932 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-15 02:55:49,933 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-15 02:55:49,934 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-15 02:55:49,934 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-15 02:55:49,934 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-15 02:55:49,934 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-15 02:55:49,934 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-15 02:55:49,934 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-15 02:55:49,934 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:49,934 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:55:49,935 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:49,935 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-15 02:55:49,935 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-15 02:55:49,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-15 02:55:49,935 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:49,935 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-15 02:55:49,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:55:49,936 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:49,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-15 02:55:49,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:55:49,936 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-15 02:55:49,936 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-15 02:55:49,936 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-15 02:55:49,936 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-15 02:55:49,936 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-15 02:55:49,936 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-15 02:55:49,936 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-15 02:55:49,936 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-15 02:55:49,936 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681527331158.a29fa882a9e3f50c391e3aa08c5c5e72. 2023-04-15 02:55:49,936 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681527331158.a29fa882a9e3f50c391e3aa08c5c5e72. started... 2023-04-15 02:55:49,937 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681527331158.a29fa882a9e3f50c391e3aa08c5c5e72. as already flushing 2023-04-15 02:55:49,937 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681527331158.a29fa882a9e3f50c391e3aa08c5c5e72. 2023-04-15 02:55:49,937 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-15 02:55:49,937 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-15 02:55:49,937 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:49,937 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-15 02:55:49,937 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34681,1681527209133' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-15 02:55:49,937 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:49,937 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:49,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:49,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:55:49,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:55:49,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:55:49,938 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:49,938 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-15 02:55:49,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:55:49,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:55:49,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:49,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:55:49,939 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34681,1681527209133' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-15 02:55:49,939 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-15 02:55:49,939 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:49,939 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7cc4c796[Count = 0] remaining members to acquire global barrier 2023-04-15 02:55:49,939 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:49,939 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:49,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:49,940 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:49,940 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-15 02:55:49,940 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-15 02:55:49,940 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-15 02:55:49,940 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34681,1681527209133' in zk 2023-04-15 02:55:49,940 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-15 02:55:49,940 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-15 02:55:49,940 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:49,940 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-15 02:55:49,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:49,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:55:49,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:55:49,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:55:49,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:55:49,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:55:49,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:49,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:55:49,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:55:49,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:49,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34681,1681527209133': 2023-04-15 02:55:49,942 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34681,1681527209133' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-15 02:55:49,942 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-15 02:55:49,942 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-15 02:55:49,942 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-15 02:55:49,942 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-15 02:55:49,942 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-15 02:55:49,943 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:49,943 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:49,943 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-15 02:55:49,943 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:49,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:49,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:49,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:55:49,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:55:49,944 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-15 02:55:49,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-15 02:55:49,944 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:49,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:55:49,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:49,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:55:49,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:55:49,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:55:49,944 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:49,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:49,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:55:49,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:55:49,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:49,946 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:49,946 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:55:49,946 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:55:49,946 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:49,946 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-15 02:55:49,946 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-15 02:55:49,946 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-15 02:55:49,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-15 02:55:49,946 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-15 02:55:49,946 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:49,946 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:49,946 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:49,946 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-15 02:55:49,946 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-15 02:55:49,946 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-15 02:55:49,946 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-15 02:55:49,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-15 02:55:49,946 DEBUG [Thread-2739] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-15 02:55:49,946 DEBUG [Thread-2739] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-15 02:55:50,035 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a29fa882a9e3f50c391e3aa08c5c5e72, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:55:50,036 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34681] ipc.CallRunner(144): callId: 216 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:40234 deadline: 1681527410031, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a29fa882a9e3f50c391e3aa08c5c5e72, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:50,036 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a29fa882a9e3f50c391e3aa08c5c5e72, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:55:50,036 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34681] ipc.CallRunner(144): callId: 214 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:40210 deadline: 1681527410034, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a29fa882a9e3f50c391e3aa08c5c5e72, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:50,036 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a29fa882a9e3f50c391e3aa08c5c5e72, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:55:50,036 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34681] ipc.CallRunner(144): callId: 213 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:40192 deadline: 1681527410034, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a29fa882a9e3f50c391e3aa08c5c5e72, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:50,047 DEBUG [Thread-2739] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-15 02:55:50,047 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-15 02:55:50,048 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-15 02:55:50,049 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-15 02:55:50,049 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-15 02:55:50,049 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-15 02:55:50,049 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-15 02:55:50,049 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-15 02:55:50,049 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-15 02:55:50,049 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:50,049 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:55:50,050 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-15 02:55:50,050 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:50,050 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-15 02:55:50,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-15 02:55:50,050 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:50,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:55:50,050 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-15 02:55:50,050 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:50,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-15 02:55:50,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:55:50,051 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-15 02:55:50,051 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-15 02:55:50,051 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-15 02:55:50,051 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-15 02:55:50,051 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-15 02:55:50,051 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-15 02:55:50,051 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-15 02:55:50,051 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-15 02:55:50,051 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681527331158.a29fa882a9e3f50c391e3aa08c5c5e72. 2023-04-15 02:55:50,051 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681527331158.a29fa882a9e3f50c391e3aa08c5c5e72. started... 2023-04-15 02:55:50,051 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681527331158.a29fa882a9e3f50c391e3aa08c5c5e72. as already flushing 2023-04-15 02:55:50,051 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681527331158.a29fa882a9e3f50c391e3aa08c5c5e72. 2023-04-15 02:55:50,051 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-15 02:55:50,051 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-15 02:55:50,051 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:50,051 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-15 02:55:50,052 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34681,1681527209133' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-15 02:55:50,052 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:50,052 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:50,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:50,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:55:50,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:55:50,052 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:50,052 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-15 02:55:50,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:55:50,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:55:50,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:55:50,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:50,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:55:50,053 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34681,1681527209133' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-15 02:55:50,054 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-15 02:55:50,054 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4f13ead5[Count = 0] remaining members to acquire global barrier 2023-04-15 02:55:50,054 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:50,054 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:50,054 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:50,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:50,054 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-15 02:55:50,054 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-15 02:55:50,054 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34681,1681527209133' in zk 2023-04-15 02:55:50,054 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:50,054 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-15 02:55:50,055 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:50,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:50,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:55:50,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:55:50,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:55:50,055 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-15 02:55:50,055 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-15 02:55:50,056 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-15 02:55:50,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:55:50,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:55:50,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:50,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:55:50,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:55:50,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:50,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34681,1681527209133': 2023-04-15 02:55:50,057 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34681,1681527209133' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-15 02:55:50,057 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-15 02:55:50,057 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-15 02:55:50,057 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-15 02:55:50,057 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-15 02:55:50,057 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-15 02:55:50,058 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:50,058 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:50,058 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:50,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:50,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:55:50,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:55:50,058 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-15 02:55:50,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:50,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:55:50,058 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:50,058 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-15 02:55:50,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-15 02:55:50,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:55:50,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:50,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:55:50,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:55:50,059 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:50,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:50,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:55:50,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:55:50,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:50,060 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:50,060 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:55:50,060 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:55:50,060 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-15 02:55:50,060 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-15 02:55:50,060 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:50,060 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-15 02:55:50,060 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-15 02:55:50,060 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-15 02:55:50,060 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-15 02:55:50,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-15 02:55:50,060 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:50,061 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:50,061 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:50,060 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-15 02:55:50,060 DEBUG [Thread-2739] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-15 02:55:50,061 DEBUG [Thread-2739] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-15 02:55:50,061 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-15 02:55:50,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-15 02:55:50,073 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=410 (bloomFilter=true), to=hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/a29fa882a9e3f50c391e3aa08c5c5e72/.tmp/A/b9c42cf4ab0b40199b787473de2b3b3d 2023-04-15 02:55:50,161 DEBUG [Thread-2739] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-15 02:55:50,161 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-15 02:55:50,163 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-15 02:55:50,163 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-15 02:55:50,163 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-15 02:55:50,163 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-15 02:55:50,163 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-15 02:55:50,164 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-15 02:55:50,164 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-15 02:55:50,164 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:50,164 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:55:50,164 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:50,165 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-15 02:55:50,165 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-15 02:55:50,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-15 02:55:50,165 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:50,165 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-15 02:55:50,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:55:50,165 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:50,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-15 02:55:50,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:55:50,165 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-15 02:55:50,165 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-15 02:55:50,165 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-15 02:55:50,165 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-15 02:55:50,165 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-15 02:55:50,165 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-15 02:55:50,166 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-15 02:55:50,166 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-15 02:55:50,166 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681527331158.a29fa882a9e3f50c391e3aa08c5c5e72. 2023-04-15 02:55:50,166 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681527331158.a29fa882a9e3f50c391e3aa08c5c5e72. started... 2023-04-15 02:55:50,166 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681527331158.a29fa882a9e3f50c391e3aa08c5c5e72. as already flushing 2023-04-15 02:55:50,166 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681527331158.a29fa882a9e3f50c391e3aa08c5c5e72. 2023-04-15 02:55:50,166 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-15 02:55:50,166 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-15 02:55:50,166 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:50,166 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-15 02:55:50,166 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34681,1681527209133' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-15 02:55:50,167 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:50,167 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:50,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:50,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:55:50,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:55:50,167 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:50,167 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-15 02:55:50,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:55:50,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:55:50,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:55:50,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:50,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:55:50,168 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34681,1681527209133' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-15 02:55:50,168 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3cc40953[Count = 0] remaining members to acquire global barrier 2023-04-15 02:55:50,168 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-15 02:55:50,168 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:50,169 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:50,169 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:50,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:50,169 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-15 02:55:50,169 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:50,169 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-15 02:55:50,169 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-15 02:55:50,169 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34681,1681527209133' in zk 2023-04-15 02:55:50,170 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-15 02:55:50,170 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:50,170 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-15 02:55:50,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:50,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:55:50,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:55:50,170 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-15 02:55:50,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:55:50,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:55:50,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:55:50,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:50,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:55:50,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:55:50,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:50,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34681,1681527209133': 2023-04-15 02:55:50,171 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34681,1681527209133' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-15 02:55:50,171 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-15 02:55:50,171 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-15 02:55:50,171 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-15 02:55:50,171 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-15 02:55:50,171 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-15 02:55:50,172 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:50,172 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:50,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:50,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:55:50,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:55:50,172 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-15 02:55:50,172 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:50,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:50,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:55:50,172 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-15 02:55:50,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-15 02:55:50,173 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:50,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:55:50,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:50,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:55:50,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:55:50,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:50,173 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:50,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:55:50,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:55:50,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:50,176 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-15 02:55:50,176 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:50,176 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-15 02:55:50,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-15 02:55:50,176 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-15 02:55:50,176 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-15 02:55:50,176 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-15 02:55:50,176 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:55:50,176 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-15 02:55:50,176 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:55:50,176 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-15 02:55:50,176 DEBUG [Thread-2739] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-15 02:55:50,176 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-15 02:55:50,176 DEBUG [Thread-2739] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-15 02:55:50,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-15 02:55:50,176 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:50,176 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:50,176 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:50,176 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:50,276 DEBUG [Thread-2739] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-15 02:55:50,277 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-15 02:55:50,278 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-15 02:55:50,279 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-15 02:55:50,279 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-15 02:55:50,279 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-15 02:55:50,279 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-15 02:55:50,279 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-15 02:55:50,279 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-15 02:55:50,279 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:50,280 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:55:50,280 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-15 02:55:50,280 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:50,280 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-15 02:55:50,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-15 02:55:50,280 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:50,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:55:50,280 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-15 02:55:50,281 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:50,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-15 02:55:50,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:55:50,281 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-15 02:55:50,281 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-15 02:55:50,281 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-15 02:55:50,281 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-15 02:55:50,281 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-15 02:55:50,281 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-15 02:55:50,281 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-15 02:55:50,281 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-15 02:55:50,281 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681527331158.a29fa882a9e3f50c391e3aa08c5c5e72. 2023-04-15 02:55:50,281 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681527331158.a29fa882a9e3f50c391e3aa08c5c5e72. started... 2023-04-15 02:55:50,281 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681527331158.a29fa882a9e3f50c391e3aa08c5c5e72. as already flushing 2023-04-15 02:55:50,281 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681527331158.a29fa882a9e3f50c391e3aa08c5c5e72. 2023-04-15 02:55:50,282 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-15 02:55:50,282 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-15 02:55:50,282 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:50,282 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-15 02:55:50,282 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34681,1681527209133' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-15 02:55:50,282 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:50,282 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:50,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:50,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:55:50,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:55:50,282 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:50,283 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-15 02:55:50,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:55:50,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:55:50,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:55:50,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:50,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:55:50,284 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34681,1681527209133' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-15 02:55:50,284 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-15 02:55:50,284 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4a0f24dd[Count = 0] remaining members to acquire global barrier 2023-04-15 02:55:50,284 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:50,284 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:50,284 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:50,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:50,284 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-15 02:55:50,284 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-15 02:55:50,284 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:50,284 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-15 02:55:50,284 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34681,1681527209133' in zk 2023-04-15 02:55:50,285 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:50,285 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-15 02:55:50,285 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-15 02:55:50,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:50,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:55:50,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:55:50,285 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-15 02:55:50,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:55:50,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:55:50,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:55:50,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:50,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:55:50,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:55:50,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:50,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34681,1681527209133': 2023-04-15 02:55:50,287 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34681,1681527209133' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-15 02:55:50,287 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-15 02:55:50,287 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-15 02:55:50,287 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-15 02:55:50,287 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-15 02:55:50,287 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-15 02:55:50,288 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:50,288 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:50,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:50,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:55:50,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:55:50,288 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:50,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:50,288 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-15 02:55:50,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:55:50,289 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-15 02:55:50,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-15 02:55:50,289 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:50,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:55:50,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:50,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:55:50,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:55:50,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:50,289 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:50,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:55:50,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:55:50,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:50,291 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-15 02:55:50,291 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-15 02:55:50,291 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-15 02:55:50,291 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:50,291 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-15 02:55:50,291 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:55:50,291 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-15 02:55:50,291 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-15 02:55:50,291 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-15 02:55:50,291 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:55:50,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-15 02:55:50,291 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:50,291 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:50,291 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:50,291 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:50,291 DEBUG [Thread-2739] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-15 02:55:50,291 DEBUG [Thread-2739] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-15 02:55:50,291 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-15 02:55:50,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-15 02:55:50,341 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a29fa882a9e3f50c391e3aa08c5c5e72, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:55:50,341 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34681] ipc.CallRunner(144): callId: 218 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:40234 deadline: 1681527410337, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a29fa882a9e3f50c391e3aa08c5c5e72, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:50,341 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a29fa882a9e3f50c391e3aa08c5c5e72, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:55:50,341 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34681] ipc.CallRunner(144): callId: 216 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:40210 deadline: 1681527410337, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a29fa882a9e3f50c391e3aa08c5c5e72, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:50,343 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a29fa882a9e3f50c391e3aa08c5c5e72, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:55:50,343 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34681] ipc.CallRunner(144): callId: 215 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:40192 deadline: 1681527410338, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a29fa882a9e3f50c391e3aa08c5c5e72, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:50,391 DEBUG [Thread-2739] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-15 02:55:50,392 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-15 02:55:50,393 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-15 02:55:50,394 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-15 02:55:50,394 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-15 02:55:50,394 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-15 02:55:50,394 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-15 02:55:50,394 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-15 02:55:50,394 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-15 02:55:50,395 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:50,395 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:55:50,395 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-15 02:55:50,395 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:50,395 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-15 02:55:50,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-15 02:55:50,395 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:50,396 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-15 02:55:50,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:55:50,396 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:50,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-15 02:55:50,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:55:50,396 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-15 02:55:50,396 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-15 02:55:50,396 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-15 02:55:50,396 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-15 02:55:50,396 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-15 02:55:50,397 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-15 02:55:50,397 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-15 02:55:50,397 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-15 02:55:50,397 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681527331158.a29fa882a9e3f50c391e3aa08c5c5e72. 2023-04-15 02:55:50,397 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681527331158.a29fa882a9e3f50c391e3aa08c5c5e72. started... 2023-04-15 02:55:50,397 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681527331158.a29fa882a9e3f50c391e3aa08c5c5e72. as already flushing 2023-04-15 02:55:50,397 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681527331158.a29fa882a9e3f50c391e3aa08c5c5e72. 2023-04-15 02:55:50,397 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-15 02:55:50,397 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-15 02:55:50,397 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:50,397 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-15 02:55:50,397 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34681,1681527209133' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-15 02:55:50,398 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:50,398 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:50,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:50,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:55:50,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:55:50,398 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:50,398 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-15 02:55:50,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:55:50,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:55:50,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:55:50,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:50,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:55:50,399 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34681,1681527209133' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-15 02:55:50,399 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-15 02:55:50,399 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@10ada27d[Count = 0] remaining members to acquire global barrier 2023-04-15 02:55:50,399 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:50,400 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:50,400 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:50,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:50,400 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:50,400 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-15 02:55:50,400 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-15 02:55:50,400 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-15 02:55:50,400 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34681,1681527209133' in zk 2023-04-15 02:55:50,401 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:50,401 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-15 02:55:50,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:50,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:55:50,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:55:50,401 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-15 02:55:50,401 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-15 02:55:50,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:55:50,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:55:50,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:55:50,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:50,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:55:50,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:55:50,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:50,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34681,1681527209133': 2023-04-15 02:55:50,403 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34681,1681527209133' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-15 02:55:50,403 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-15 02:55:50,403 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-15 02:55:50,403 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-15 02:55:50,403 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-15 02:55:50,403 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-15 02:55:50,403 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:50,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:50,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:55:50,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:55:50,404 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:50,404 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-15 02:55:50,404 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:50,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:50,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:55:50,404 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-15 02:55:50,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-15 02:55:50,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:50,404 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:50,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:55:50,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:55:50,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:55:50,405 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:50,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:50,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:55:50,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:55:50,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:50,407 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:50,407 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:55:50,407 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:55:50,407 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-15 02:55:50,407 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-15 02:55:50,407 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-15 02:55:50,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-15 02:55:50,407 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-15 02:55:50,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-15 02:55:50,408 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:50,408 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:50,408 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:50,408 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-15 02:55:50,408 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-15 02:55:50,408 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-15 02:55:50,408 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:50,408 DEBUG [Thread-2739] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-15 02:55:50,408 DEBUG [Thread-2739] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-15 02:55:50,409 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-15 02:55:50,483 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=410 (bloomFilter=true), to=hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/a29fa882a9e3f50c391e3aa08c5c5e72/.tmp/B/5aa162be14cf46b997e28b721c0920aa 2023-04-15 02:55:50,509 DEBUG [Thread-2739] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-15 02:55:50,509 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-15 02:55:50,510 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-15 02:55:50,511 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-15 02:55:50,511 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-15 02:55:50,511 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-15 02:55:50,511 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-15 02:55:50,512 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-15 02:55:50,512 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-15 02:55:50,512 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:50,512 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:55:50,513 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-15 02:55:50,513 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-15 02:55:50,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-15 02:55:50,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:55:50,513 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:50,513 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:50,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-15 02:55:50,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:55:50,513 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-15 02:55:50,513 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:50,514 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-15 02:55:50,514 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-15 02:55:50,514 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-15 02:55:50,514 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-15 02:55:50,514 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-15 02:55:50,514 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-15 02:55:50,514 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-15 02:55:50,514 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-15 02:55:50,514 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681527331158.a29fa882a9e3f50c391e3aa08c5c5e72. 2023-04-15 02:55:50,514 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681527331158.a29fa882a9e3f50c391e3aa08c5c5e72. started... 2023-04-15 02:55:50,514 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681527331158.a29fa882a9e3f50c391e3aa08c5c5e72. as already flushing 2023-04-15 02:55:50,514 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681527331158.a29fa882a9e3f50c391e3aa08c5c5e72. 2023-04-15 02:55:50,514 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-15 02:55:50,514 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-15 02:55:50,514 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:50,514 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-15 02:55:50,514 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34681,1681527209133' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-15 02:55:50,515 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:50,515 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:50,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:50,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:55:50,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:55:50,515 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:50,515 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-15 02:55:50,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:55:50,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:55:50,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:55:50,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:50,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:55:50,517 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34681,1681527209133' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-15 02:55:50,517 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-15 02:55:50,517 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4038a7ff[Count = 0] remaining members to acquire global barrier 2023-04-15 02:55:50,517 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:50,517 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:50,517 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:50,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:50,517 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-15 02:55:50,517 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-15 02:55:50,517 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34681,1681527209133' in zk 2023-04-15 02:55:50,518 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:50,518 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-15 02:55:50,519 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-15 02:55:50,519 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-15 02:55:50,519 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-15 02:55:50,519 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:50,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:50,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:55:50,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:55:50,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:55:50,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:55:50,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:55:50,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:50,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:55:50,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:55:50,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:50,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34681,1681527209133': 2023-04-15 02:55:50,522 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34681,1681527209133' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-15 02:55:50,522 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-15 02:55:50,522 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-15 02:55:50,523 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-15 02:55:50,523 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-15 02:55:50,523 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-15 02:55:50,523 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:50,523 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:50,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:50,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:55:50,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:55:50,523 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:50,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:50,523 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-15 02:55:50,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:55:50,524 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-15 02:55:50,524 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:50,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-15 02:55:50,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:55:50,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:55:50,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:50,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:55:50,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:50,525 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:50,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:55:50,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:55:50,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:50,526 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-15 02:55:50,526 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-15 02:55:50,526 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-15 02:55:50,526 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-15 02:55:50,526 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-15 02:55:50,526 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-15 02:55:50,526 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:50,526 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:55:50,526 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:55:50,526 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:50,526 DEBUG [Thread-2739] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-15 02:55:50,526 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-15 02:55:50,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-15 02:55:50,527 DEBUG [Thread-2739] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-15 02:55:50,526 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:50,527 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:50,527 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:50,527 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-15 02:55:50,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-15 02:55:50,627 DEBUG [Thread-2739] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-15 02:55:50,627 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-15 02:55:50,628 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-15 02:55:50,629 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-15 02:55:50,629 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-15 02:55:50,629 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-15 02:55:50,629 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-15 02:55:50,629 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-15 02:55:50,629 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-15 02:55:50,629 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:50,629 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:55:50,630 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:50,630 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-15 02:55:50,630 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-15 02:55:50,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-15 02:55:50,630 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:50,630 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-15 02:55:50,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:55:50,631 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:50,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-15 02:55:50,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:55:50,631 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-15 02:55:50,631 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-15 02:55:50,631 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-15 02:55:50,631 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-15 02:55:50,631 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-15 02:55:50,631 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-15 02:55:50,631 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-15 02:55:50,631 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-15 02:55:50,631 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681527331158.a29fa882a9e3f50c391e3aa08c5c5e72. 2023-04-15 02:55:50,631 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681527331158.a29fa882a9e3f50c391e3aa08c5c5e72. started... 2023-04-15 02:55:50,631 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681527331158.a29fa882a9e3f50c391e3aa08c5c5e72. as already flushing 2023-04-15 02:55:50,631 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681527331158.a29fa882a9e3f50c391e3aa08c5c5e72. 2023-04-15 02:55:50,632 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-15 02:55:50,632 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-15 02:55:50,632 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:50,632 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-15 02:55:50,632 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34681,1681527209133' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-15 02:55:50,632 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:50,632 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:50,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:50,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:55:50,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:55:50,633 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:50,633 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-15 02:55:50,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:55:50,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:55:50,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:55:50,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:50,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:55:50,633 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34681,1681527209133' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-15 02:55:50,633 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-15 02:55:50,633 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5c2aa03a[Count = 0] remaining members to acquire global barrier 2023-04-15 02:55:50,634 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:50,634 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:50,634 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:50,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:50,634 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-15 02:55:50,634 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-15 02:55:50,634 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34681,1681527209133' in zk 2023-04-15 02:55:50,635 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:50,635 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-15 02:55:50,636 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:50,636 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-15 02:55:50,636 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-15 02:55:50,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:50,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:55:50,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:55:50,636 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-15 02:55:50,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:55:50,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:55:50,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:55:50,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:50,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:55:50,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:55:50,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:50,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34681,1681527209133': 2023-04-15 02:55:50,637 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34681,1681527209133' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-15 02:55:50,637 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-15 02:55:50,637 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-15 02:55:50,637 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-15 02:55:50,637 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-15 02:55:50,638 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-15 02:55:50,638 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:50,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:50,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:55:50,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:55:50,638 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:50,638 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-15 02:55:50,638 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:50,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:50,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:55:50,639 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-15 02:55:50,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-15 02:55:50,639 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:50,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:55:50,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:50,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:55:50,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:55:50,640 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:50,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:50,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:55:50,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:55:50,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:50,641 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:50,641 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-15 02:55:50,641 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:55:50,641 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-15 02:55:50,641 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-15 02:55:50,641 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-15 02:55:50,641 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-15 02:55:50,641 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:55:50,641 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:50,641 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:50,641 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:50,641 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:50,641 DEBUG [Thread-2739] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-15 02:55:50,641 DEBUG [Thread-2739] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-15 02:55:50,641 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-15 02:55:50,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-15 02:55:50,641 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-15 02:55:50,642 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-15 02:55:50,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-15 02:55:50,741 DEBUG [Thread-2739] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-15 02:55:50,742 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-15 02:55:50,743 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-15 02:55:50,744 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-15 02:55:50,744 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-15 02:55:50,744 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-15 02:55:50,744 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-15 02:55:50,744 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-15 02:55:50,744 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-15 02:55:50,744 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:50,744 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:55:50,745 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-15 02:55:50,745 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-15 02:55:50,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-15 02:55:50,745 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:50,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:55:50,745 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:50,745 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-15 02:55:50,745 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:50,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-15 02:55:50,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:55:50,746 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-15 02:55:50,746 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-15 02:55:50,746 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-15 02:55:50,746 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-15 02:55:50,746 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-15 02:55:50,746 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-15 02:55:50,746 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-15 02:55:50,746 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-15 02:55:50,746 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681527331158.a29fa882a9e3f50c391e3aa08c5c5e72. 2023-04-15 02:55:50,746 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681527331158.a29fa882a9e3f50c391e3aa08c5c5e72. started... 2023-04-15 02:55:50,746 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681527331158.a29fa882a9e3f50c391e3aa08c5c5e72. as already flushing 2023-04-15 02:55:50,746 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681527331158.a29fa882a9e3f50c391e3aa08c5c5e72. 2023-04-15 02:55:50,746 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-15 02:55:50,746 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-15 02:55:50,746 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:50,746 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-15 02:55:50,746 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34681,1681527209133' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-15 02:55:50,747 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:50,747 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:50,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:50,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:55:50,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:55:50,747 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:50,747 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-15 02:55:50,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:55:50,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:55:50,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:55:50,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:50,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:55:50,748 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34681,1681527209133' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-15 02:55:50,749 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-15 02:55:50,749 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6f304eae[Count = 0] remaining members to acquire global barrier 2023-04-15 02:55:50,749 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:50,749 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:50,749 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:50,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:50,749 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-15 02:55:50,749 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-15 02:55:50,749 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:50,749 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-15 02:55:50,749 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34681,1681527209133' in zk 2023-04-15 02:55:50,750 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:50,750 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-15 02:55:50,750 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-15 02:55:50,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:50,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:55:50,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:55:50,750 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-15 02:55:50,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:55:50,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:55:50,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:55:50,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:50,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:55:50,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:55:50,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:50,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34681,1681527209133': 2023-04-15 02:55:50,752 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34681,1681527209133' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-15 02:55:50,752 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-15 02:55:50,752 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-15 02:55:50,752 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-15 02:55:50,752 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-15 02:55:50,752 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-15 02:55:50,753 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:50,753 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:50,753 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-15 02:55:50,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:50,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:55:50,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:55:50,753 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:50,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:50,753 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:50,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:55:50,753 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-15 02:55:50,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-15 02:55:50,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:55:50,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:50,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:55:50,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:55:50,754 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:50,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:50,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:55:50,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:55:50,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:50,755 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:50,755 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-15 02:55:50,755 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:55:50,755 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-15 02:55:50,755 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:55:50,755 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-15 02:55:50,755 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-15 02:55:50,755 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-15 02:55:50,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-15 02:55:50,755 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-15 02:55:50,755 DEBUG [Thread-2739] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-15 02:55:50,755 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-15 02:55:50,755 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:50,756 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-15 02:55:50,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-15 02:55:50,756 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:50,755 DEBUG [Thread-2739] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-15 02:55:50,756 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:50,756 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:50,844 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a29fa882a9e3f50c391e3aa08c5c5e72, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:55:50,844 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34681] ipc.CallRunner(144): callId: 220 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:40234 deadline: 1681527410842, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a29fa882a9e3f50c391e3aa08c5c5e72, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:50,844 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a29fa882a9e3f50c391e3aa08c5c5e72, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:55:50,845 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34681] ipc.CallRunner(144): callId: 218 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:40210 deadline: 1681527410843, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a29fa882a9e3f50c391e3aa08c5c5e72, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:50,851 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a29fa882a9e3f50c391e3aa08c5c5e72, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:55:50,851 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34681] ipc.CallRunner(144): callId: 217 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:40192 deadline: 1681527410848, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a29fa882a9e3f50c391e3aa08c5c5e72, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:50,856 DEBUG [Thread-2739] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-15 02:55:50,856 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-15 02:55:50,858 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-15 02:55:50,858 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-15 02:55:50,858 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-15 02:55:50,858 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-15 02:55:50,858 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-15 02:55:50,858 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-15 02:55:50,859 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-15 02:55:50,859 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:50,859 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:55:50,859 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:50,859 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-15 02:55:50,859 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-15 02:55:50,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-15 02:55:50,860 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:50,860 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-15 02:55:50,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:55:50,860 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:50,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-15 02:55:50,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:55:50,860 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-15 02:55:50,860 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-15 02:55:50,860 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-15 02:55:50,860 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-15 02:55:50,860 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-15 02:55:50,860 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-15 02:55:50,861 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-15 02:55:50,861 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-15 02:55:50,861 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681527331158.a29fa882a9e3f50c391e3aa08c5c5e72. 2023-04-15 02:55:50,861 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681527331158.a29fa882a9e3f50c391e3aa08c5c5e72. started... 2023-04-15 02:55:50,861 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681527331158.a29fa882a9e3f50c391e3aa08c5c5e72. as already flushing 2023-04-15 02:55:50,861 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681527331158.a29fa882a9e3f50c391e3aa08c5c5e72. 2023-04-15 02:55:50,861 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-15 02:55:50,861 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-15 02:55:50,861 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:50,861 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-15 02:55:50,861 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34681,1681527209133' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-15 02:55:50,862 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:50,862 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:50,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:50,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:55:50,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:55:50,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:55:50,862 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:50,862 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-15 02:55:50,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:55:50,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:55:50,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:50,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:55:50,864 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34681,1681527209133' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-15 02:55:50,864 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@12f3b77f[Count = 0] remaining members to acquire global barrier 2023-04-15 02:55:50,864 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-15 02:55:50,864 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:50,865 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:50,865 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:50,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:50,865 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-15 02:55:50,865 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-15 02:55:50,865 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34681,1681527209133' in zk 2023-04-15 02:55:50,865 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:50,865 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-15 02:55:50,865 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:50,865 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-15 02:55:50,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:50,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:55:50,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:55:50,865 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-15 02:55:50,866 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-15 02:55:50,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:55:50,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:55:50,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:55:50,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:50,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:55:50,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:55:50,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:50,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34681,1681527209133': 2023-04-15 02:55:50,867 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34681,1681527209133' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-15 02:55:50,867 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-15 02:55:50,867 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-15 02:55:50,867 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-15 02:55:50,867 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-15 02:55:50,868 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-15 02:55:50,868 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:50,868 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:50,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:50,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:55:50,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:55:50,868 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-15 02:55:50,868 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:50,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:50,869 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-15 02:55:50,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-15 02:55:50,869 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:50,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:55:50,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:50,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:55:50,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:55:50,870 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:50,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:55:50,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:50,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:55:50,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:55:50,871 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:50,872 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:55:50,872 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-15 02:55:50,872 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-15 02:55:50,872 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:55:50,872 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-15 02:55:50,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-15 02:55:50,872 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-15 02:55:50,872 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:50,872 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:50,872 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-15 02:55:50,872 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:50,872 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:50,872 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-15 02:55:50,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-15 02:55:50,872 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-15 02:55:50,872 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-15 02:55:50,873 DEBUG [Thread-2739] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-15 02:55:50,873 DEBUG [Thread-2739] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-15 02:55:50,892 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=410 (bloomFilter=true), to=hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/a29fa882a9e3f50c391e3aa08c5c5e72/.tmp/C/d69e5dbe9ea14e0c876d82eb7d904f65 2023-04-15 02:55:50,895 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/a29fa882a9e3f50c391e3aa08c5c5e72/.tmp/A/b9c42cf4ab0b40199b787473de2b3b3d as hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/a29fa882a9e3f50c391e3aa08c5c5e72/A/b9c42cf4ab0b40199b787473de2b3b3d 2023-04-15 02:55:50,898 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/a29fa882a9e3f50c391e3aa08c5c5e72/A/b9c42cf4ab0b40199b787473de2b3b3d, entries=250, sequenceid=410, filesize=16.7 K 2023-04-15 02:55:50,898 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/a29fa882a9e3f50c391e3aa08c5c5e72/.tmp/B/5aa162be14cf46b997e28b721c0920aa as hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/a29fa882a9e3f50c391e3aa08c5c5e72/B/5aa162be14cf46b997e28b721c0920aa 2023-04-15 02:55:50,901 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/a29fa882a9e3f50c391e3aa08c5c5e72/B/5aa162be14cf46b997e28b721c0920aa, entries=150, sequenceid=410, filesize=11.9 K 2023-04-15 02:55:50,901 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/a29fa882a9e3f50c391e3aa08c5c5e72/.tmp/C/d69e5dbe9ea14e0c876d82eb7d904f65 as hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/a29fa882a9e3f50c391e3aa08c5c5e72/C/d69e5dbe9ea14e0c876d82eb7d904f65 2023-04-15 02:55:50,904 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/a29fa882a9e3f50c391e3aa08c5c5e72/C/d69e5dbe9ea14e0c876d82eb7d904f65, entries=150, sequenceid=410, filesize=11.9 K 2023-04-15 02:55:50,905 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=134.18 KB/137400 for a29fa882a9e3f50c391e3aa08c5c5e72 in 1238ms, sequenceid=410, compaction requested=true 2023-04-15 02:55:50,905 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for a29fa882a9e3f50c391e3aa08c5c5e72: 2023-04-15 02:55:50,905 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-15 02:55:50,905 DEBUG [RS:0;jenkins-hbase20:34681-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-15 02:55:50,905 DEBUG [RS:0;jenkins-hbase20:34681-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-15 02:55:50,905 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-15 02:55:50,905 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-15 02:55:50,910 DEBUG [RS:0;jenkins-hbase20:34681-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37342 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-15 02:55:50,910 DEBUG [RS:0;jenkins-hbase20:34681-longCompactions-0] regionserver.HStore(1912): a29fa882a9e3f50c391e3aa08c5c5e72/B is initiating minor compaction (all files) 2023-04-15 02:55:50,910 INFO [RS:0;jenkins-hbase20:34681-longCompactions-0] regionserver.HRegion(2259): Starting compaction of a29fa882a9e3f50c391e3aa08c5c5e72/B in TestAcidGuarantees,,1681527331158.a29fa882a9e3f50c391e3aa08c5c5e72. 2023-04-15 02:55:50,910 INFO [RS:0;jenkins-hbase20:34681-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/a29fa882a9e3f50c391e3aa08c5c5e72/B/558068e1a6fd477385b4c9b7a02a5be1, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/a29fa882a9e3f50c391e3aa08c5c5e72/B/513f80e4f4a141dc97f9f4db67f57ba6, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/a29fa882a9e3f50c391e3aa08c5c5e72/B/5aa162be14cf46b997e28b721c0920aa] into tmpdir=hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/a29fa882a9e3f50c391e3aa08c5c5e72/.tmp, totalSize=36.5 K 2023-04-15 02:55:50,911 DEBUG [RS:0;jenkins-hbase20:34681-longCompactions-0] compactions.Compactor(207): Compacting 558068e1a6fd477385b4c9b7a02a5be1, keycount=150, bloomtype=ROW, size=12.7 K, encoding=NONE, compression=NONE, seqNum=374, earliestPutTs=1681527346880 2023-04-15 02:55:50,911 DEBUG [RS:0;jenkins-hbase20:34681-longCompactions-0] compactions.Compactor(207): Compacting 513f80e4f4a141dc97f9f4db67f57ba6, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=395, earliestPutTs=1681527347510 2023-04-15 02:55:50,911 DEBUG [RS:0;jenkins-hbase20:34681-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 42222 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-15 02:55:50,911 DEBUG [RS:0;jenkins-hbase20:34681-shortCompactions-0] regionserver.HStore(1912): a29fa882a9e3f50c391e3aa08c5c5e72/A is initiating minor compaction (all files) 2023-04-15 02:55:50,911 INFO [RS:0;jenkins-hbase20:34681-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of a29fa882a9e3f50c391e3aa08c5c5e72/A in TestAcidGuarantees,,1681527331158.a29fa882a9e3f50c391e3aa08c5c5e72. 2023-04-15 02:55:50,911 DEBUG [RS:0;jenkins-hbase20:34681-longCompactions-0] compactions.Compactor(207): Compacting 5aa162be14cf46b997e28b721c0920aa, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=410, earliestPutTs=1681527349661 2023-04-15 02:55:50,911 INFO [RS:0;jenkins-hbase20:34681-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/a29fa882a9e3f50c391e3aa08c5c5e72/A/8b168677d78840c7bc3dba0bfc7fa4cc, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/a29fa882a9e3f50c391e3aa08c5c5e72/A/90e4b2984b0b4171b9e59884e699b38b, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/a29fa882a9e3f50c391e3aa08c5c5e72/A/b9c42cf4ab0b40199b787473de2b3b3d] into tmpdir=hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/a29fa882a9e3f50c391e3aa08c5c5e72/.tmp, totalSize=41.2 K 2023-04-15 02:55:50,912 DEBUG [RS:0;jenkins-hbase20:34681-shortCompactions-0] compactions.Compactor(207): Compacting 8b168677d78840c7bc3dba0bfc7fa4cc, keycount=150, bloomtype=ROW, size=12.7 K, encoding=NONE, compression=NONE, seqNum=374, earliestPutTs=1681527346880 2023-04-15 02:55:50,912 DEBUG [RS:0;jenkins-hbase20:34681-shortCompactions-0] compactions.Compactor(207): Compacting 90e4b2984b0b4171b9e59884e699b38b, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=395, earliestPutTs=1681527347510 2023-04-15 02:55:50,912 DEBUG [RS:0;jenkins-hbase20:34681-shortCompactions-0] compactions.Compactor(207): Compacting b9c42cf4ab0b40199b787473de2b3b3d, keycount=250, bloomtype=ROW, size=16.7 K, encoding=NONE, compression=NONE, seqNum=410, earliestPutTs=1681527349657 2023-04-15 02:55:50,917 INFO [RS:0;jenkins-hbase20:34681-longCompactions-0] throttle.PressureAwareThroughputController(145): a29fa882a9e3f50c391e3aa08c5c5e72#B#compaction#422 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-15 02:55:50,918 INFO [RS:0;jenkins-hbase20:34681-shortCompactions-0] throttle.PressureAwareThroughputController(145): a29fa882a9e3f50c391e3aa08c5c5e72#A#compaction#423 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-15 02:55:50,935 DEBUG [RS:0;jenkins-hbase20:34681-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/a29fa882a9e3f50c391e3aa08c5c5e72/.tmp/B/728653d7917c45ad8e9b5fe14c5a1cd1 as hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/a29fa882a9e3f50c391e3aa08c5c5e72/B/728653d7917c45ad8e9b5fe14c5a1cd1 2023-04-15 02:55:50,939 INFO [RS:0;jenkins-hbase20:34681-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in a29fa882a9e3f50c391e3aa08c5c5e72/B of a29fa882a9e3f50c391e3aa08c5c5e72 into 728653d7917c45ad8e9b5fe14c5a1cd1(size=12.8 K), total size for store is 12.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-15 02:55:50,939 DEBUG [RS:0;jenkins-hbase20:34681-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for a29fa882a9e3f50c391e3aa08c5c5e72: 2023-04-15 02:55:50,939 INFO [RS:0;jenkins-hbase20:34681-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681527331158.a29fa882a9e3f50c391e3aa08c5c5e72., storeName=a29fa882a9e3f50c391e3aa08c5c5e72/B, priority=13, startTime=1681527350905; duration=0sec 2023-04-15 02:55:50,939 DEBUG [RS:0;jenkins-hbase20:34681-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-15 02:55:50,939 DEBUG [RS:0;jenkins-hbase20:34681-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-15 02:55:50,940 DEBUG [RS:0;jenkins-hbase20:34681-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37342 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-15 02:55:50,940 DEBUG [RS:0;jenkins-hbase20:34681-longCompactions-0] regionserver.HStore(1912): a29fa882a9e3f50c391e3aa08c5c5e72/C is initiating minor compaction (all files) 2023-04-15 02:55:50,940 INFO [RS:0;jenkins-hbase20:34681-longCompactions-0] regionserver.HRegion(2259): Starting compaction of a29fa882a9e3f50c391e3aa08c5c5e72/C in TestAcidGuarantees,,1681527331158.a29fa882a9e3f50c391e3aa08c5c5e72. 2023-04-15 02:55:50,940 INFO [RS:0;jenkins-hbase20:34681-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/a29fa882a9e3f50c391e3aa08c5c5e72/C/35cbbdf3250541bb914fe4d9c46f472c, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/a29fa882a9e3f50c391e3aa08c5c5e72/C/35d587b695f74c159d812876bf3420cc, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/a29fa882a9e3f50c391e3aa08c5c5e72/C/d69e5dbe9ea14e0c876d82eb7d904f65] into tmpdir=hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/a29fa882a9e3f50c391e3aa08c5c5e72/.tmp, totalSize=36.5 K 2023-04-15 02:55:50,940 DEBUG [RS:0;jenkins-hbase20:34681-longCompactions-0] compactions.Compactor(207): Compacting 35cbbdf3250541bb914fe4d9c46f472c, keycount=150, bloomtype=ROW, size=12.7 K, encoding=NONE, compression=NONE, seqNum=374, earliestPutTs=1681527346880 2023-04-15 02:55:50,941 DEBUG [RS:0;jenkins-hbase20:34681-longCompactions-0] compactions.Compactor(207): Compacting 35d587b695f74c159d812876bf3420cc, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=395, earliestPutTs=1681527347510 2023-04-15 02:55:50,941 DEBUG [RS:0;jenkins-hbase20:34681-longCompactions-0] compactions.Compactor(207): Compacting d69e5dbe9ea14e0c876d82eb7d904f65, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=410, earliestPutTs=1681527349661 2023-04-15 02:55:50,946 INFO [RS:0;jenkins-hbase20:34681-longCompactions-0] throttle.PressureAwareThroughputController(145): a29fa882a9e3f50c391e3aa08c5c5e72#C#compaction#424 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-15 02:55:50,973 DEBUG [Thread-2739] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-15 02:55:50,973 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-15 02:55:50,974 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-15 02:55:50,975 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-15 02:55:50,975 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-15 02:55:50,975 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-15 02:55:50,975 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-15 02:55:50,975 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-15 02:55:50,975 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-15 02:55:50,976 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:50,976 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:55:50,976 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-15 02:55:50,976 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:50,977 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-15 02:55:50,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-15 02:55:50,977 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:50,977 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-15 02:55:50,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:55:50,977 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:50,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-15 02:55:50,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:55:50,978 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-15 02:55:50,978 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-15 02:55:50,978 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-15 02:55:50,978 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-15 02:55:50,978 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-15 02:55:50,978 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-15 02:55:50,978 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-15 02:55:50,978 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-15 02:55:50,978 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681527331158.a29fa882a9e3f50c391e3aa08c5c5e72. 2023-04-15 02:55:50,979 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681527331158.a29fa882a9e3f50c391e3aa08c5c5e72. started... 2023-04-15 02:55:50,980 INFO [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing a29fa882a9e3f50c391e3aa08c5c5e72 3/3 column families, dataSize=134.18 KB heapSize=380.30 KB 2023-04-15 02:55:50,980 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a29fa882a9e3f50c391e3aa08c5c5e72, store=A 2023-04-15 02:55:50,980 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-15 02:55:50,980 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a29fa882a9e3f50c391e3aa08c5c5e72, store=B 2023-04-15 02:55:50,980 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-15 02:55:50,980 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a29fa882a9e3f50c391e3aa08c5c5e72, store=C 2023-04-15 02:55:50,980 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-15 02:55:50,994 INFO [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=447 (bloomFilter=true), to=hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/a29fa882a9e3f50c391e3aa08c5c5e72/.tmp/A/9a723c5189444482886f596aa7488555 2023-04-15 02:55:51,337 DEBUG [RS:0;jenkins-hbase20:34681-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/a29fa882a9e3f50c391e3aa08c5c5e72/.tmp/A/69c91ab709194d56b46e176b3215d30a as hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/a29fa882a9e3f50c391e3aa08c5c5e72/A/69c91ab709194d56b46e176b3215d30a 2023-04-15 02:55:51,340 INFO [RS:0;jenkins-hbase20:34681-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in a29fa882a9e3f50c391e3aa08c5c5e72/A of a29fa882a9e3f50c391e3aa08c5c5e72 into 69c91ab709194d56b46e176b3215d30a(size=12.8 K), total size for store is 12.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-15 02:55:51,340 DEBUG [RS:0;jenkins-hbase20:34681-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for a29fa882a9e3f50c391e3aa08c5c5e72: 2023-04-15 02:55:51,340 INFO [RS:0;jenkins-hbase20:34681-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681527331158.a29fa882a9e3f50c391e3aa08c5c5e72., storeName=a29fa882a9e3f50c391e3aa08c5c5e72/A, priority=13, startTime=1681527350905; duration=0sec 2023-04-15 02:55:51,341 DEBUG [RS:0;jenkins-hbase20:34681-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-15 02:55:51,361 DEBUG [RS:0;jenkins-hbase20:34681-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/a29fa882a9e3f50c391e3aa08c5c5e72/.tmp/C/44bb03c191ed4eb5ae0c07767164cc4e as hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/a29fa882a9e3f50c391e3aa08c5c5e72/C/44bb03c191ed4eb5ae0c07767164cc4e 2023-04-15 02:55:51,365 INFO [RS:0;jenkins-hbase20:34681-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in a29fa882a9e3f50c391e3aa08c5c5e72/C of a29fa882a9e3f50c391e3aa08c5c5e72 into 44bb03c191ed4eb5ae0c07767164cc4e(size=12.8 K), total size for store is 12.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-15 02:55:51,365 DEBUG [RS:0;jenkins-hbase20:34681-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for a29fa882a9e3f50c391e3aa08c5c5e72: 2023-04-15 02:55:51,365 INFO [RS:0;jenkins-hbase20:34681-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681527331158.a29fa882a9e3f50c391e3aa08c5c5e72., storeName=a29fa882a9e3f50c391e3aa08c5c5e72/C, priority=13, startTime=1681527350905; duration=0sec 2023-04-15 02:55:51,365 DEBUG [RS:0;jenkins-hbase20:34681-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-15 02:55:51,401 INFO [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=447 (bloomFilter=true), to=hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/a29fa882a9e3f50c391e3aa08c5c5e72/.tmp/B/53153d4147c2459c959a071addb308ab 2023-04-15 02:55:51,811 INFO [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=447 (bloomFilter=true), to=hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/a29fa882a9e3f50c391e3aa08c5c5e72/.tmp/C/b672c6f4c74140069d562b1d2bf96131 2023-04-15 02:55:51,814 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/a29fa882a9e3f50c391e3aa08c5c5e72/.tmp/A/9a723c5189444482886f596aa7488555 as hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/a29fa882a9e3f50c391e3aa08c5c5e72/A/9a723c5189444482886f596aa7488555 2023-04-15 02:55:51,817 INFO [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/a29fa882a9e3f50c391e3aa08c5c5e72/A/9a723c5189444482886f596aa7488555, entries=150, sequenceid=447, filesize=11.9 K 2023-04-15 02:55:51,817 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/a29fa882a9e3f50c391e3aa08c5c5e72/.tmp/B/53153d4147c2459c959a071addb308ab as hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/a29fa882a9e3f50c391e3aa08c5c5e72/B/53153d4147c2459c959a071addb308ab 2023-04-15 02:55:51,820 INFO [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/a29fa882a9e3f50c391e3aa08c5c5e72/B/53153d4147c2459c959a071addb308ab, entries=150, sequenceid=447, filesize=11.9 K 2023-04-15 02:55:51,821 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/a29fa882a9e3f50c391e3aa08c5c5e72/.tmp/C/b672c6f4c74140069d562b1d2bf96131 as hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/a29fa882a9e3f50c391e3aa08c5c5e72/C/b672c6f4c74140069d562b1d2bf96131 2023-04-15 02:55:51,823 INFO [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/a29fa882a9e3f50c391e3aa08c5c5e72/C/b672c6f4c74140069d562b1d2bf96131, entries=150, sequenceid=447, filesize=11.9 K 2023-04-15 02:55:51,824 INFO [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~134.18 KB/137400, heapSize ~380.25 KB/389376, currentSize=0 B/0 for a29fa882a9e3f50c391e3aa08c5c5e72 in 844ms, sequenceid=447, compaction requested=false 2023-04-15 02:55:51,824 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for a29fa882a9e3f50c391e3aa08c5c5e72: 2023-04-15 02:55:51,824 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681527331158.a29fa882a9e3f50c391e3aa08c5c5e72. 2023-04-15 02:55:51,824 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-15 02:55:51,824 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-15 02:55:51,824 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:51,824 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-15 02:55:51,824 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34681,1681527209133' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-15 02:55:51,826 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:51,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:51,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:55:51,826 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:51,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:55:51,827 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:51,827 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-15 02:55:51,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:55:51,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:55:51,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:55:51,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:51,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:55:51,828 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34681,1681527209133' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-15 02:55:51,828 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@17f1ed81[Count = 0] remaining members to acquire global barrier 2023-04-15 02:55:51,828 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-15 02:55:51,828 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:51,828 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:51,828 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:51,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:51,828 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-15 02:55:51,828 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-15 02:55:51,828 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:51,828 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-15 02:55:51,828 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34681,1681527209133' in zk 2023-04-15 02:55:51,829 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-15 02:55:51,829 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-15 02:55:51,829 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-15 02:55:51,829 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:51,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:51,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:55:51,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:55:51,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:55:51,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:55:51,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:55:51,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:51,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:55:51,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:55:51,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:51,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34681,1681527209133': 2023-04-15 02:55:51,832 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34681,1681527209133' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-15 02:55:51,832 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-15 02:55:51,832 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-15 02:55:51,832 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-15 02:55:51,832 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-15 02:55:51,832 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-15 02:55:51,833 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:51,833 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:51,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:51,833 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:51,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:51,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:55:51,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:55:51,833 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-15 02:55:51,833 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-15 02:55:51,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-15 02:55:51,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:51,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:55:51,834 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:51,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:55:51,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:55:51,835 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:51,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:55:51,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:51,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:55:51,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:55:51,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:51,837 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:51,837 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:55:51,837 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:55:51,837 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:51,837 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:51,837 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-15 02:55:51,837 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-15 02:55:51,837 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:51,837 DEBUG [Thread-2739] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-15 02:55:51,837 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-15 02:55:51,837 DEBUG [Thread-2739] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-15 02:55:51,837 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:51,837 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-15 02:55:51,837 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-15 02:55:51,837 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-15 02:55:51,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-15 02:55:51,837 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-15 02:55:51,838 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-15 02:55:51,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-15 02:55:51,856 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34681] regionserver.HRegion(9158): Flush requested on a29fa882a9e3f50c391e3aa08c5c5e72 2023-04-15 02:55:51,856 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing a29fa882a9e3f50c391e3aa08c5c5e72 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-15 02:55:51,856 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a29fa882a9e3f50c391e3aa08c5c5e72, store=A 2023-04-15 02:55:51,856 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-15 02:55:51,856 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a29fa882a9e3f50c391e3aa08c5c5e72, store=B 2023-04-15 02:55:51,856 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-15 02:55:51,857 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a29fa882a9e3f50c391e3aa08c5c5e72, store=C 2023-04-15 02:55:51,857 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-15 02:55:51,888 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=309, cellSize=14145, totalHeapSize=32376, min timestamp=1681527351879, max timestamp=1681527351887Num uniques 859; 2023-04-15 02:55:51,888 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-9] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=14145, suffix heap size=117064,new segment heap size=32376 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=309 2023-04-15 02:55:51,889 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=309, cellSize=14145, totalHeapSize=32376, min timestamp=1681527351879, max timestamp=1681527351887Num uniques 859; 2023-04-15 02:55:51,889 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=309, cellSize=14145, totalHeapSize=32376, min timestamp=1681527351879, max timestamp=1681527351887Num uniques 859; 2023-04-15 02:55:51,889 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-3] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=14145, suffix heap size=117064,new segment heap size=32376 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=309 2023-04-15 02:55:51,889 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-5] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=14145, suffix heap size=117064,new segment heap size=32376 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=309 2023-04-15 02:55:51,937 DEBUG [Thread-2739] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-15 02:55:51,938 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-15 02:55:51,939 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-15 02:55:51,939 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-15 02:55:51,939 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-15 02:55:51,939 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-15 02:55:51,939 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-15 02:55:51,939 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-15 02:55:51,940 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-15 02:55:51,940 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:51,940 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:55:51,940 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-15 02:55:51,940 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:51,940 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-15 02:55:51,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-15 02:55:51,941 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:51,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:55:51,941 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-15 02:55:51,941 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:51,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-15 02:55:51,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:55:51,941 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-15 02:55:51,941 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-15 02:55:51,942 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-15 02:55:51,942 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-15 02:55:51,942 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-15 02:55:51,942 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-15 02:55:51,942 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-15 02:55:51,942 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-15 02:55:51,942 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681527331158.a29fa882a9e3f50c391e3aa08c5c5e72. 2023-04-15 02:55:51,942 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681527331158.a29fa882a9e3f50c391e3aa08c5c5e72. started... 2023-04-15 02:55:51,942 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681527331158.a29fa882a9e3f50c391e3aa08c5c5e72. as already flushing 2023-04-15 02:55:51,942 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681527331158.a29fa882a9e3f50c391e3aa08c5c5e72. 2023-04-15 02:55:51,942 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-15 02:55:51,942 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-15 02:55:51,942 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:51,942 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-15 02:55:51,942 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34681,1681527209133' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-15 02:55:51,943 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:51,943 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:51,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:51,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:55:51,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:55:51,943 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:51,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:55:51,943 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-15 02:55:51,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:55:51,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:55:51,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:51,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:55:51,945 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34681,1681527209133' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-15 02:55:51,945 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-15 02:55:51,945 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1ed3b435[Count = 0] remaining members to acquire global barrier 2023-04-15 02:55:51,945 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:51,945 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:51,945 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:51,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:51,945 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-15 02:55:51,945 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-15 02:55:51,945 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:51,945 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-15 02:55:51,945 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34681,1681527209133' in zk 2023-04-15 02:55:51,946 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:51,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:51,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:55:51,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:55:51,946 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-15 02:55:51,946 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-15 02:55:51,946 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-15 02:55:51,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:55:51,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:55:51,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:55:51,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:51,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:55:51,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:55:51,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:51,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34681,1681527209133': 2023-04-15 02:55:51,948 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34681,1681527209133' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-15 02:55:51,948 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-15 02:55:51,948 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-15 02:55:51,948 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-15 02:55:51,948 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-15 02:55:51,948 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-15 02:55:51,949 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:51,949 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:51,949 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-15 02:55:51,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:51,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:55:51,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:55:51,949 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:51,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:51,949 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-15 02:55:51,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-15 02:55:51,949 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:51,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:55:51,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:51,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:55:51,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:55:51,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:55:51,950 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a29fa882a9e3f50c391e3aa08c5c5e72, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:55:51,950 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34681] ipc.CallRunner(144): callId: 231 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:40210 deadline: 1681527411944, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a29fa882a9e3f50c391e3aa08c5c5e72, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:51,950 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:51,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:51,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:55:51,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:55:51,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:51,951 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:51,951 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:55:51,952 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:55:51,951 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-15 02:55:51,952 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-15 02:55:51,952 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-15 02:55:51,952 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-15 02:55:51,952 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:51,952 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-15 02:55:51,952 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-15 02:55:51,952 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-15 02:55:51,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-15 02:55:51,952 DEBUG [Thread-2739] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-15 02:55:51,952 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:51,952 DEBUG [Thread-2739] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-15 02:55:51,952 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:51,952 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:51,952 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-15 02:55:51,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-15 02:55:51,953 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a29fa882a9e3f50c391e3aa08c5c5e72, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:55:51,953 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34681] ipc.CallRunner(144): callId: 229 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:40192 deadline: 1681527411945, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a29fa882a9e3f50c391e3aa08c5c5e72, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:51,953 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a29fa882a9e3f50c391e3aa08c5c5e72, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:55:51,953 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34681] ipc.CallRunner(144): callId: 240 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:40234 deadline: 1681527411946, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a29fa882a9e3f50c391e3aa08c5c5e72, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:52,052 DEBUG [Thread-2739] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-15 02:55:52,053 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-15 02:55:52,053 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a29fa882a9e3f50c391e3aa08c5c5e72, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:55:52,053 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34681] ipc.CallRunner(144): callId: 233 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:40210 deadline: 1681527412051, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a29fa882a9e3f50c391e3aa08c5c5e72, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:52,054 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-15 02:55:52,054 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-15 02:55:52,054 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-15 02:55:52,054 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-15 02:55:52,055 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-15 02:55:52,055 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-15 02:55:52,055 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-15 02:55:52,055 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:52,055 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:55:52,055 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-15 02:55:52,055 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:52,055 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-15 02:55:52,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-15 02:55:52,056 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:52,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:55:52,056 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-15 02:55:52,056 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:52,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-15 02:55:52,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:55:52,056 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-15 02:55:52,056 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-15 02:55:52,057 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-15 02:55:52,057 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-15 02:55:52,057 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-15 02:55:52,057 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-15 02:55:52,057 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-15 02:55:52,057 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-15 02:55:52,057 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681527331158.a29fa882a9e3f50c391e3aa08c5c5e72. 2023-04-15 02:55:52,057 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681527331158.a29fa882a9e3f50c391e3aa08c5c5e72. started... 2023-04-15 02:55:52,057 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681527331158.a29fa882a9e3f50c391e3aa08c5c5e72. as already flushing 2023-04-15 02:55:52,057 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681527331158.a29fa882a9e3f50c391e3aa08c5c5e72. 2023-04-15 02:55:52,057 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-15 02:55:52,057 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-15 02:55:52,057 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:52,057 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-15 02:55:52,057 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34681,1681527209133' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-15 02:55:52,057 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a29fa882a9e3f50c391e3aa08c5c5e72, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:55:52,057 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34681] ipc.CallRunner(144): callId: 231 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:40192 deadline: 1681527412053, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a29fa882a9e3f50c391e3aa08c5c5e72, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:52,058 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a29fa882a9e3f50c391e3aa08c5c5e72, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:55:52,058 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34681] ipc.CallRunner(144): callId: 242 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:40234 deadline: 1681527412053, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a29fa882a9e3f50c391e3aa08c5c5e72, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:52,058 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:52,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:52,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:55:52,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:55:52,058 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:52,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:55:52,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:55:52,059 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:52,059 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-15 02:55:52,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:55:52,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:52,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:55:52,059 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34681,1681527209133' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-15 02:55:52,059 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@737bcc7f[Count = 0] remaining members to acquire global barrier 2023-04-15 02:55:52,059 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-15 02:55:52,059 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:52,060 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:52,060 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:52,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:52,060 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-15 02:55:52,060 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-15 02:55:52,060 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34681,1681527209133' in zk 2023-04-15 02:55:52,060 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:52,060 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-15 02:55:52,061 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:52,061 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-15 02:55:52,061 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-15 02:55:52,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:52,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:55:52,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:55:52,061 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-15 02:55:52,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:55:52,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:55:52,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:55:52,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:52,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:55:52,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:55:52,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:52,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34681,1681527209133': 2023-04-15 02:55:52,063 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34681,1681527209133' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-15 02:55:52,063 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-15 02:55:52,063 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-15 02:55:52,063 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-15 02:55:52,063 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-15 02:55:52,063 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-15 02:55:52,064 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:52,064 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-15 02:55:52,064 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:52,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:52,064 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:52,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:52,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:55:52,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:55:52,064 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-15 02:55:52,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-15 02:55:52,064 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:52,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:55:52,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:52,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:55:52,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:55:52,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:55:52,065 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:52,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:52,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:55:52,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:55:52,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:52,067 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:52,067 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:55:52,067 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-15 02:55:52,067 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:55:52,067 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-15 02:55:52,067 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-15 02:55:52,067 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-15 02:55:52,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-15 02:55:52,067 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-15 02:55:52,067 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-15 02:55:52,067 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:52,067 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-15 02:55:52,067 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-15 02:55:52,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-15 02:55:52,067 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:52,067 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:52,067 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:52,067 DEBUG [Thread-2739] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-15 02:55:52,068 DEBUG [Thread-2739] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-15 02:55:52,168 DEBUG [Thread-2739] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-15 02:55:52,168 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-15 02:55:52,169 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-15 02:55:52,170 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-15 02:55:52,170 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-15 02:55:52,170 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-15 02:55:52,170 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-15 02:55:52,170 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-15 02:55:52,170 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-15 02:55:52,171 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:52,171 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:55:52,171 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:52,171 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-15 02:55:52,171 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-15 02:55:52,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-15 02:55:52,171 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:52,171 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-15 02:55:52,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:55:52,172 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:52,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-15 02:55:52,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:55:52,172 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-15 02:55:52,173 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-15 02:55:52,173 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-15 02:55:52,173 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-15 02:55:52,173 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-15 02:55:52,173 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-15 02:55:52,173 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-15 02:55:52,173 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-15 02:55:52,173 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681527331158.a29fa882a9e3f50c391e3aa08c5c5e72. 2023-04-15 02:55:52,173 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681527331158.a29fa882a9e3f50c391e3aa08c5c5e72. started... 2023-04-15 02:55:52,173 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681527331158.a29fa882a9e3f50c391e3aa08c5c5e72. as already flushing 2023-04-15 02:55:52,173 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681527331158.a29fa882a9e3f50c391e3aa08c5c5e72. 2023-04-15 02:55:52,173 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-15 02:55:52,173 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-15 02:55:52,173 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:52,173 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-15 02:55:52,173 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34681,1681527209133' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-15 02:55:52,174 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:52,174 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:52,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:52,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:55:52,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:55:52,174 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:52,174 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-15 02:55:52,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:55:52,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:55:52,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:55:52,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:52,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:55:52,175 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34681,1681527209133' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-15 02:55:52,175 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-15 02:55:52,175 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7cbcd520[Count = 0] remaining members to acquire global barrier 2023-04-15 02:55:52,175 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:52,176 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:52,176 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:52,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:52,176 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-15 02:55:52,176 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-15 02:55:52,176 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34681,1681527209133' in zk 2023-04-15 02:55:52,177 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-15 02:55:52,177 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-15 02:55:52,177 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-15 02:55:52,177 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:52,178 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34681,1681527209133' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-15 02:55:52,178 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-15 02:55:52,178 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-15 02:55:52,178 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-15 02:55:52,178 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-15 02:55:52,178 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-15 02:55:52,178 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-15 02:55:52,178 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:52,178 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:52,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:52,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:55:52,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:55:52,178 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:52,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:52,178 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-15 02:55:52,179 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-15 02:55:52,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:55:52,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-15 02:55:52,179 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:52,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:55:52,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:52,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:55:52,180 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:52,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:55:52,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:52,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:55:52,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:55:52,183 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-15 02:55:52,183 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-15 02:55:52,183 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:52,183 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:55:52,183 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:55:52,183 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-15 02:55:52,183 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-15 02:55:52,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-15 02:55:52,183 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:52,183 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-15 02:55:52,183 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-15 02:55:52,183 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-15 02:55:52,183 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-15 02:55:52,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-15 02:55:52,183 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:52,184 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:52,184 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:52,184 DEBUG [Thread-2739] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-15 02:55:52,184 DEBUG [Thread-2739] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-15 02:55:52,257 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a29fa882a9e3f50c391e3aa08c5c5e72, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:55:52,257 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34681] ipc.CallRunner(144): callId: 235 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:40210 deadline: 1681527412254, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a29fa882a9e3f50c391e3aa08c5c5e72, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:52,262 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=459 (bloomFilter=true), to=hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/a29fa882a9e3f50c391e3aa08c5c5e72/.tmp/A/efcbca754cc14ecdb9f9a36e94daa51e 2023-04-15 02:55:52,263 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a29fa882a9e3f50c391e3aa08c5c5e72, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:55:52,263 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34681] ipc.CallRunner(144): callId: 244 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:40234 deadline: 1681527412258, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a29fa882a9e3f50c391e3aa08c5c5e72, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:52,263 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a29fa882a9e3f50c391e3aa08c5c5e72, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:55:52,263 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34681] ipc.CallRunner(144): callId: 233 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:40192 deadline: 1681527412259, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a29fa882a9e3f50c391e3aa08c5c5e72, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:52,284 DEBUG [Thread-2739] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-15 02:55:52,284 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-15 02:55:52,286 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-15 02:55:52,287 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-15 02:55:52,287 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-15 02:55:52,287 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-15 02:55:52,287 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-15 02:55:52,287 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-15 02:55:52,287 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-15 02:55:52,289 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:52,289 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:55:52,290 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-15 02:55:52,290 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:52,290 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-15 02:55:52,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-15 02:55:52,290 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:52,290 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-15 02:55:52,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:55:52,290 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:52,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-15 02:55:52,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:55:52,291 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-15 02:55:52,291 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-15 02:55:52,291 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-15 02:55:52,291 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-15 02:55:52,291 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-15 02:55:52,291 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-15 02:55:52,291 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-15 02:55:52,291 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-15 02:55:52,291 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681527331158.a29fa882a9e3f50c391e3aa08c5c5e72. 2023-04-15 02:55:52,291 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681527331158.a29fa882a9e3f50c391e3aa08c5c5e72. started... 2023-04-15 02:55:52,291 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681527331158.a29fa882a9e3f50c391e3aa08c5c5e72. as already flushing 2023-04-15 02:55:52,291 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681527331158.a29fa882a9e3f50c391e3aa08c5c5e72. 2023-04-15 02:55:52,291 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-15 02:55:52,291 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-15 02:55:52,291 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:52,291 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-15 02:55:52,291 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34681,1681527209133' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-15 02:55:52,292 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:52,292 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:52,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:52,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:55:52,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:55:52,292 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:52,292 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-15 02:55:52,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:55:52,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:55:52,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:55:52,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:52,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:55:52,293 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34681,1681527209133' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-15 02:55:52,293 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@64b9a2d3[Count = 0] remaining members to acquire global barrier 2023-04-15 02:55:52,293 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-15 02:55:52,293 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:52,294 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:52,294 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:52,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:52,294 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-15 02:55:52,294 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-15 02:55:52,294 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:52,294 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-15 02:55:52,294 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34681,1681527209133' in zk 2023-04-15 02:55:52,295 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:52,295 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-15 02:55:52,295 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-15 02:55:52,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:52,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:55:52,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:55:52,295 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-15 02:55:52,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:55:52,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:55:52,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:55:52,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:52,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:55:52,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:55:52,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:52,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34681,1681527209133': 2023-04-15 02:55:52,297 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34681,1681527209133' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-15 02:55:52,297 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-15 02:55:52,297 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-15 02:55:52,297 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-15 02:55:52,297 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-15 02:55:52,297 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-15 02:55:52,298 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:52,298 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:52,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:52,298 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:52,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:52,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:55:52,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:55:52,298 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-15 02:55:52,298 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:52,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:55:52,298 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-15 02:55:52,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-15 02:55:52,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:55:52,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:52,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:55:52,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:55:52,299 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:52,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:52,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:55:52,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:55:52,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:52,300 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:52,300 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:55:52,300 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:55:52,300 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-15 02:55:52,300 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:52,300 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-15 02:55:52,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-15 02:55:52,300 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-15 02:55:52,300 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-15 02:55:52,300 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-15 02:55:52,301 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-15 02:55:52,300 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-15 02:55:52,300 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:52,301 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:52,301 DEBUG [Thread-2739] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-15 02:55:52,301 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-15 02:55:52,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-15 02:55:52,301 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:52,301 DEBUG [Thread-2739] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-15 02:55:52,379 DEBUG [Thread-2746] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x4ea9d51a to 127.0.0.1:51762 2023-04-15 02:55:52,379 DEBUG [Thread-2740] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x41ccedcb to 127.0.0.1:51762 2023-04-15 02:55:52,379 DEBUG [Thread-2746] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-15 02:55:52,379 DEBUG [Thread-2740] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-15 02:55:52,381 DEBUG [Thread-2748] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x31fd1c1f to 127.0.0.1:51762 2023-04-15 02:55:52,381 DEBUG [Thread-2748] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-15 02:55:52,385 DEBUG [Thread-2744] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x2a5d7fcf to 127.0.0.1:51762 2023-04-15 02:55:52,385 DEBUG [Thread-2744] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-15 02:55:52,386 DEBUG [Thread-2742] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x40e6f624 to 127.0.0.1:51762 2023-04-15 02:55:52,386 DEBUG [Thread-2742] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-15 02:55:52,401 DEBUG [Thread-2739] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-15 02:55:52,401 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-15 02:55:52,560 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a29fa882a9e3f50c391e3aa08c5c5e72, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:55:52,561 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34681] ipc.CallRunner(144): callId: 237 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:40210 deadline: 1681527412560, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a29fa882a9e3f50c391e3aa08c5c5e72, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:52,565 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a29fa882a9e3f50c391e3aa08c5c5e72, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:55:52,565 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34681] ipc.CallRunner(144): callId: 235 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:40192 deadline: 1681527412565, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a29fa882a9e3f50c391e3aa08c5c5e72, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:52,566 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a29fa882a9e3f50c391e3aa08c5c5e72, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:55:52,566 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34681] ipc.CallRunner(144): callId: 246 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:40234 deadline: 1681527412566, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a29fa882a9e3f50c391e3aa08c5c5e72, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:52,670 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=459 (bloomFilter=true), to=hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/a29fa882a9e3f50c391e3aa08c5c5e72/.tmp/B/5540a2e518584131ba9305c4dbf6db1a 2023-04-15 02:55:53,064 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a29fa882a9e3f50c391e3aa08c5c5e72, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:55:53,064 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34681] ipc.CallRunner(144): callId: 239 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:40210 deadline: 1681527413064, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a29fa882a9e3f50c391e3aa08c5c5e72, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:53,069 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a29fa882a9e3f50c391e3aa08c5c5e72, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:55:53,069 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34681] ipc.CallRunner(144): callId: 237 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:40192 deadline: 1681527413069, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a29fa882a9e3f50c391e3aa08c5c5e72, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:53,070 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a29fa882a9e3f50c391e3aa08c5c5e72, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:55:53,070 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34681] ipc.CallRunner(144): callId: 248 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:40234 deadline: 1681527413070, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a29fa882a9e3f50c391e3aa08c5c5e72, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:53,092 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=459 (bloomFilter=true), to=hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/a29fa882a9e3f50c391e3aa08c5c5e72/.tmp/C/2334d8d638eb47ddb44efe91c62993b2 2023-04-15 02:55:53,101 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/a29fa882a9e3f50c391e3aa08c5c5e72/.tmp/A/efcbca754cc14ecdb9f9a36e94daa51e as hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/a29fa882a9e3f50c391e3aa08c5c5e72/A/efcbca754cc14ecdb9f9a36e94daa51e 2023-04-15 02:55:53,110 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/a29fa882a9e3f50c391e3aa08c5c5e72/A/efcbca754cc14ecdb9f9a36e94daa51e, entries=150, sequenceid=459, filesize=11.9 K 2023-04-15 02:55:53,111 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/a29fa882a9e3f50c391e3aa08c5c5e72/.tmp/B/5540a2e518584131ba9305c4dbf6db1a as hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/a29fa882a9e3f50c391e3aa08c5c5e72/B/5540a2e518584131ba9305c4dbf6db1a 2023-04-15 02:55:53,117 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/a29fa882a9e3f50c391e3aa08c5c5e72/B/5540a2e518584131ba9305c4dbf6db1a, entries=150, sequenceid=459, filesize=11.9 K 2023-04-15 02:55:53,119 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/a29fa882a9e3f50c391e3aa08c5c5e72/.tmp/C/2334d8d638eb47ddb44efe91c62993b2 as hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/a29fa882a9e3f50c391e3aa08c5c5e72/C/2334d8d638eb47ddb44efe91c62993b2 2023-04-15 02:55:53,124 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/a29fa882a9e3f50c391e3aa08c5c5e72/C/2334d8d638eb47ddb44efe91c62993b2, entries=150, sequenceid=459, filesize=11.9 K 2023-04-15 02:55:53,125 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=140.89 KB/144270 for a29fa882a9e3f50c391e3aa08c5c5e72 in 1269ms, sequenceid=459, compaction requested=true 2023-04-15 02:55:53,125 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for a29fa882a9e3f50c391e3aa08c5c5e72: 2023-04-15 02:55:53,125 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-15 02:55:53,125 DEBUG [RS:0;jenkins-hbase20:34681-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-15 02:55:53,125 DEBUG [RS:0;jenkins-hbase20:34681-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-15 02:55:53,125 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-15 02:55:53,125 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-15 02:55:53,126 DEBUG [RS:0;jenkins-hbase20:34681-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37444 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-15 02:55:53,126 DEBUG [RS:0;jenkins-hbase20:34681-shortCompactions-0] regionserver.HStore(1912): a29fa882a9e3f50c391e3aa08c5c5e72/A is initiating minor compaction (all files) 2023-04-15 02:55:53,126 DEBUG [RS:0;jenkins-hbase20:34681-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37444 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-15 02:55:53,126 INFO [RS:0;jenkins-hbase20:34681-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of a29fa882a9e3f50c391e3aa08c5c5e72/A in TestAcidGuarantees,,1681527331158.a29fa882a9e3f50c391e3aa08c5c5e72. 2023-04-15 02:55:53,126 DEBUG [RS:0;jenkins-hbase20:34681-longCompactions-0] regionserver.HStore(1912): a29fa882a9e3f50c391e3aa08c5c5e72/B is initiating minor compaction (all files) 2023-04-15 02:55:53,126 INFO [RS:0;jenkins-hbase20:34681-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/a29fa882a9e3f50c391e3aa08c5c5e72/A/69c91ab709194d56b46e176b3215d30a, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/a29fa882a9e3f50c391e3aa08c5c5e72/A/9a723c5189444482886f596aa7488555, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/a29fa882a9e3f50c391e3aa08c5c5e72/A/efcbca754cc14ecdb9f9a36e94daa51e] into tmpdir=hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/a29fa882a9e3f50c391e3aa08c5c5e72/.tmp, totalSize=36.6 K 2023-04-15 02:55:53,126 INFO [RS:0;jenkins-hbase20:34681-longCompactions-0] regionserver.HRegion(2259): Starting compaction of a29fa882a9e3f50c391e3aa08c5c5e72/B in TestAcidGuarantees,,1681527331158.a29fa882a9e3f50c391e3aa08c5c5e72. 2023-04-15 02:55:53,127 INFO [RS:0;jenkins-hbase20:34681-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/a29fa882a9e3f50c391e3aa08c5c5e72/B/728653d7917c45ad8e9b5fe14c5a1cd1, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/a29fa882a9e3f50c391e3aa08c5c5e72/B/53153d4147c2459c959a071addb308ab, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/a29fa882a9e3f50c391e3aa08c5c5e72/B/5540a2e518584131ba9305c4dbf6db1a] into tmpdir=hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/a29fa882a9e3f50c391e3aa08c5c5e72/.tmp, totalSize=36.6 K 2023-04-15 02:55:53,127 DEBUG [RS:0;jenkins-hbase20:34681-shortCompactions-0] compactions.Compactor(207): Compacting 69c91ab709194d56b46e176b3215d30a, keycount=150, bloomtype=ROW, size=12.8 K, encoding=NONE, compression=NONE, seqNum=410, earliestPutTs=1681527349661 2023-04-15 02:55:53,127 DEBUG [RS:0;jenkins-hbase20:34681-longCompactions-0] compactions.Compactor(207): Compacting 728653d7917c45ad8e9b5fe14c5a1cd1, keycount=150, bloomtype=ROW, size=12.8 K, encoding=NONE, compression=NONE, seqNum=410, earliestPutTs=1681527349661 2023-04-15 02:55:53,127 DEBUG [RS:0;jenkins-hbase20:34681-shortCompactions-0] compactions.Compactor(207): Compacting 9a723c5189444482886f596aa7488555, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=447, earliestPutTs=1681527349715 2023-04-15 02:55:53,127 DEBUG [RS:0;jenkins-hbase20:34681-longCompactions-0] compactions.Compactor(207): Compacting 53153d4147c2459c959a071addb308ab, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=447, earliestPutTs=1681527349715 2023-04-15 02:55:53,128 DEBUG [RS:0;jenkins-hbase20:34681-shortCompactions-0] compactions.Compactor(207): Compacting efcbca754cc14ecdb9f9a36e94daa51e, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=459, earliestPutTs=1681527351849 2023-04-15 02:55:53,128 DEBUG [RS:0;jenkins-hbase20:34681-longCompactions-0] compactions.Compactor(207): Compacting 5540a2e518584131ba9305c4dbf6db1a, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=459, earliestPutTs=1681527351849 2023-04-15 02:55:53,136 INFO [RS:0;jenkins-hbase20:34681-longCompactions-0] throttle.PressureAwareThroughputController(145): a29fa882a9e3f50c391e3aa08c5c5e72#B#compaction#431 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-15 02:55:53,136 INFO [RS:0;jenkins-hbase20:34681-shortCompactions-0] throttle.PressureAwareThroughputController(145): a29fa882a9e3f50c391e3aa08c5c5e72#A#compaction#432 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-15 02:55:53,145 DEBUG [RS:0;jenkins-hbase20:34681-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/a29fa882a9e3f50c391e3aa08c5c5e72/.tmp/A/454801a3e8174a35b153f39efae8f0cf as hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/a29fa882a9e3f50c391e3aa08c5c5e72/A/454801a3e8174a35b153f39efae8f0cf 2023-04-15 02:55:53,148 INFO [RS:0;jenkins-hbase20:34681-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in a29fa882a9e3f50c391e3aa08c5c5e72/A of a29fa882a9e3f50c391e3aa08c5c5e72 into 454801a3e8174a35b153f39efae8f0cf(size=12.9 K), total size for store is 12.9 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-15 02:55:53,148 DEBUG [RS:0;jenkins-hbase20:34681-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for a29fa882a9e3f50c391e3aa08c5c5e72: 2023-04-15 02:55:53,148 INFO [RS:0;jenkins-hbase20:34681-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681527331158.a29fa882a9e3f50c391e3aa08c5c5e72., storeName=a29fa882a9e3f50c391e3aa08c5c5e72/A, priority=13, startTime=1681527353125; duration=0sec 2023-04-15 02:55:53,149 DEBUG [RS:0;jenkins-hbase20:34681-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-15 02:55:53,149 DEBUG [RS:0;jenkins-hbase20:34681-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-15 02:55:53,149 DEBUG [RS:0;jenkins-hbase20:34681-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37444 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-15 02:55:53,149 DEBUG [RS:0;jenkins-hbase20:34681-shortCompactions-0] regionserver.HStore(1912): a29fa882a9e3f50c391e3aa08c5c5e72/C is initiating minor compaction (all files) 2023-04-15 02:55:53,149 INFO [RS:0;jenkins-hbase20:34681-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of a29fa882a9e3f50c391e3aa08c5c5e72/C in TestAcidGuarantees,,1681527331158.a29fa882a9e3f50c391e3aa08c5c5e72. 2023-04-15 02:55:53,149 INFO [RS:0;jenkins-hbase20:34681-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/a29fa882a9e3f50c391e3aa08c5c5e72/C/44bb03c191ed4eb5ae0c07767164cc4e, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/a29fa882a9e3f50c391e3aa08c5c5e72/C/b672c6f4c74140069d562b1d2bf96131, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/a29fa882a9e3f50c391e3aa08c5c5e72/C/2334d8d638eb47ddb44efe91c62993b2] into tmpdir=hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/a29fa882a9e3f50c391e3aa08c5c5e72/.tmp, totalSize=36.6 K 2023-04-15 02:55:53,150 DEBUG [RS:0;jenkins-hbase20:34681-shortCompactions-0] compactions.Compactor(207): Compacting 44bb03c191ed4eb5ae0c07767164cc4e, keycount=150, bloomtype=ROW, size=12.8 K, encoding=NONE, compression=NONE, seqNum=410, earliestPutTs=1681527349661 2023-04-15 02:55:53,150 DEBUG [RS:0;jenkins-hbase20:34681-shortCompactions-0] compactions.Compactor(207): Compacting b672c6f4c74140069d562b1d2bf96131, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=447, earliestPutTs=1681527349715 2023-04-15 02:55:53,150 DEBUG [RS:0;jenkins-hbase20:34681-shortCompactions-0] compactions.Compactor(207): Compacting 2334d8d638eb47ddb44efe91c62993b2, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=459, earliestPutTs=1681527351849 2023-04-15 02:55:53,154 INFO [RS:0;jenkins-hbase20:34681-shortCompactions-0] throttle.PressureAwareThroughputController(145): a29fa882a9e3f50c391e3aa08c5c5e72#C#compaction#433 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-15 02:55:53,546 DEBUG [RS:0;jenkins-hbase20:34681-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/a29fa882a9e3f50c391e3aa08c5c5e72/.tmp/B/6ca987c0fd8f4e8a9389deba06e5447d as hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/a29fa882a9e3f50c391e3aa08c5c5e72/B/6ca987c0fd8f4e8a9389deba06e5447d 2023-04-15 02:55:53,549 INFO [RS:0;jenkins-hbase20:34681-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in a29fa882a9e3f50c391e3aa08c5c5e72/B of a29fa882a9e3f50c391e3aa08c5c5e72 into 6ca987c0fd8f4e8a9389deba06e5447d(size=12.9 K), total size for store is 12.9 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-15 02:55:53,549 DEBUG [RS:0;jenkins-hbase20:34681-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for a29fa882a9e3f50c391e3aa08c5c5e72: 2023-04-15 02:55:53,549 INFO [RS:0;jenkins-hbase20:34681-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681527331158.a29fa882a9e3f50c391e3aa08c5c5e72., storeName=a29fa882a9e3f50c391e3aa08c5c5e72/B, priority=13, startTime=1681527353125; duration=0sec 2023-04-15 02:55:53,550 DEBUG [RS:0;jenkins-hbase20:34681-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-15 02:55:53,561 DEBUG [RS:0;jenkins-hbase20:34681-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/a29fa882a9e3f50c391e3aa08c5c5e72/.tmp/C/1ccd90366ab74a97944655816a237ecd as hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/a29fa882a9e3f50c391e3aa08c5c5e72/C/1ccd90366ab74a97944655816a237ecd 2023-04-15 02:55:53,565 INFO [RS:0;jenkins-hbase20:34681-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in a29fa882a9e3f50c391e3aa08c5c5e72/C of a29fa882a9e3f50c391e3aa08c5c5e72 into 1ccd90366ab74a97944655816a237ecd(size=12.9 K), total size for store is 12.9 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-15 02:55:53,565 DEBUG [RS:0;jenkins-hbase20:34681-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for a29fa882a9e3f50c391e3aa08c5c5e72: 2023-04-15 02:55:53,565 INFO [RS:0;jenkins-hbase20:34681-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681527331158.a29fa882a9e3f50c391e3aa08c5c5e72., storeName=a29fa882a9e3f50c391e3aa08c5c5e72/C, priority=13, startTime=1681527353125; duration=0sec 2023-04-15 02:55:53,565 DEBUG [RS:0;jenkins-hbase20:34681-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-15 02:55:54,073 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34681] regionserver.HRegion(9158): Flush requested on a29fa882a9e3f50c391e3aa08c5c5e72 2023-04-15 02:55:54,073 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing a29fa882a9e3f50c391e3aa08c5c5e72 3/3 column families, dataSize=147.60 KB heapSize=410.77 KB 2023-04-15 02:55:54,073 DEBUG [Thread-2735] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x51ff7cd4 to 127.0.0.1:51762 2023-04-15 02:55:54,073 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a29fa882a9e3f50c391e3aa08c5c5e72, store=A 2023-04-15 02:55:54,074 DEBUG [Thread-2735] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-15 02:55:54,074 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-15 02:55:54,074 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a29fa882a9e3f50c391e3aa08c5c5e72, store=B 2023-04-15 02:55:54,075 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-15 02:55:54,075 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a29fa882a9e3f50c391e3aa08c5c5e72, store=C 2023-04-15 02:55:54,075 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-15 02:55:54,078 DEBUG [Thread-2737] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x06846250 to 127.0.0.1:51762 2023-04-15 02:55:54,078 DEBUG [Thread-2737] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-15 02:55:54,078 DEBUG [Thread-2729] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x517cd407 to 127.0.0.1:51762 2023-04-15 02:55:54,078 DEBUG [Thread-2729] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-15 02:55:54,247 DEBUG [Thread-2733] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x6ea7ac79 to 127.0.0.1:51762 2023-04-15 02:55:54,247 DEBUG [Thread-2733] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-15 02:55:54,271 DEBUG [Thread-2731] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x3e895a5d to 127.0.0.1:51762 2023-04-15 02:55:54,271 DEBUG [Thread-2731] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-15 02:55:54,271 INFO [Listener at localhost.localdomain/33693] hbase.AcidGuaranteesTestTool(392): Finished test. Writers: 2023-04-15 02:55:54,271 INFO [Listener at localhost.localdomain/33693] hbase.AcidGuaranteesTestTool(394): wrote 123 2023-04-15 02:55:54,272 INFO [Listener at localhost.localdomain/33693] hbase.AcidGuaranteesTestTool(394): wrote 26 2023-04-15 02:55:54,272 INFO [Listener at localhost.localdomain/33693] hbase.AcidGuaranteesTestTool(394): wrote 27 2023-04-15 02:55:54,272 INFO [Listener at localhost.localdomain/33693] hbase.AcidGuaranteesTestTool(394): wrote 114 2023-04-15 02:55:54,272 INFO [Listener at localhost.localdomain/33693] hbase.AcidGuaranteesTestTool(394): wrote 112 2023-04-15 02:55:54,272 INFO [Listener at localhost.localdomain/33693] hbase.AcidGuaranteesTestTool(396): Readers: 2023-04-15 02:55:54,272 INFO [Listener at localhost.localdomain/33693] hbase.AcidGuaranteesTestTool(400): Scanners: 2023-04-15 02:55:54,272 INFO [Listener at localhost.localdomain/33693] hbase.AcidGuaranteesTestTool(402): scanned 2491 2023-04-15 02:55:54,272 INFO [Listener at localhost.localdomain/33693] hbase.AcidGuaranteesTestTool(403): verified 7472 rows 2023-04-15 02:55:54,272 INFO [Listener at localhost.localdomain/33693] hbase.AcidGuaranteesTestTool(402): scanned 2494 2023-04-15 02:55:54,272 INFO [Listener at localhost.localdomain/33693] hbase.AcidGuaranteesTestTool(403): verified 7482 rows 2023-04-15 02:55:54,272 INFO [Listener at localhost.localdomain/33693] hbase.AcidGuaranteesTestTool(402): scanned 2473 2023-04-15 02:55:54,272 INFO [Listener at localhost.localdomain/33693] hbase.AcidGuaranteesTestTool(403): verified 7418 rows 2023-04-15 02:55:54,272 INFO [Listener at localhost.localdomain/33693] hbase.AcidGuaranteesTestTool(402): scanned 2486 2023-04-15 02:55:54,272 INFO [Listener at localhost.localdomain/33693] hbase.AcidGuaranteesTestTool(403): verified 7457 rows 2023-04-15 02:55:54,272 INFO [Listener at localhost.localdomain/33693] hbase.AcidGuaranteesTestTool(402): scanned 2497 2023-04-15 02:55:54,273 INFO [Listener at localhost.localdomain/33693] hbase.AcidGuaranteesTestTool(403): verified 7490 rows 2023-04-15 02:55:54,273 INFO [Listener at localhost.localdomain/33693] client.ConnectionImplementation(1974): Closing master protocol: MasterService 2023-04-15 02:55:54,273 DEBUG [Listener at localhost.localdomain/33693] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x1a0b0d8a to 127.0.0.1:51762 2023-04-15 02:55:54,273 DEBUG [Listener at localhost.localdomain/33693] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-15 02:55:54,275 INFO [Listener at localhost.localdomain/33693] client.HBaseAdmin$15(890): Started disable of TestAcidGuarantees 2023-04-15 02:55:54,275 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.HMaster$11(2418): Client=jenkins//148.251.75.209 disable TestAcidGuarantees 2023-04-15 02:55:54,276 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] procedure2.ProcedureExecutor(1029): Stored pid=50, state=RUNNABLE:DISABLE_TABLE_PREPARE; DisableTableProcedure table=TestAcidGuarantees 2023-04-15 02:55:54,281 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(1227): Checking to see if procedure is done pid=50 2023-04-15 02:55:54,281 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1681527354281"}]},"ts":"1681527354281"} 2023-04-15 02:55:54,282 INFO [PEWorker-3] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=DISABLING in hbase:meta 2023-04-15 02:55:54,287 INFO [PEWorker-3] procedure.DisableTableProcedure(293): Set TestAcidGuarantees to state=DISABLING 2023-04-15 02:55:54,288 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=51, ppid=50, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=a29fa882a9e3f50c391e3aa08c5c5e72, UNASSIGN}] 2023-04-15 02:55:54,289 INFO [PEWorker-5] procedure.MasterProcedureScheduler(727): Took xlock for pid=51, ppid=50, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=a29fa882a9e3f50c391e3aa08c5c5e72, UNASSIGN 2023-04-15 02:55:54,289 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=51 updating hbase:meta row=a29fa882a9e3f50c391e3aa08c5c5e72, regionState=CLOSING, regionLocation=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:54,290 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1681527331158.a29fa882a9e3f50c391e3aa08c5c5e72.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1681527354289"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1681527354289"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1681527354289"}]},"ts":"1681527354289"} 2023-04-15 02:55:54,290 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=52, ppid=51, state=RUNNABLE; CloseRegionProcedure a29fa882a9e3f50c391e3aa08c5c5e72, server=jenkins-hbase20.apache.org,34681,1681527209133}] 2023-04-15 02:55:54,382 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(1227): Checking to see if procedure is done pid=50 2023-04-15 02:55:54,442 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] handler.UnassignRegionHandler(109): Close a29fa882a9e3f50c391e3aa08c5c5e72 2023-04-15 02:55:54,442 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1604): Closing a29fa882a9e3f50c391e3aa08c5c5e72, disabling compactions & flushes 2023-04-15 02:55:54,442 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1858): waiting for 0 compactions & cache flush to complete for region TestAcidGuarantees,,1681527331158.a29fa882a9e3f50c391e3aa08c5c5e72. 2023-04-15 02:55:54,486 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=49.20 KB at sequenceid=498 (bloomFilter=true), to=hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/a29fa882a9e3f50c391e3aa08c5c5e72/.tmp/A/6ec73ecb341c44509a59650277eb89c1 2023-04-15 02:55:54,585 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(1227): Checking to see if procedure is done pid=50 2023-04-15 02:55:54,887 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(1227): Checking to see if procedure is done pid=50 2023-04-15 02:55:54,904 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=49.20 KB at sequenceid=498 (bloomFilter=true), to=hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/a29fa882a9e3f50c391e3aa08c5c5e72/.tmp/B/ef39994cb7ec48cd9fb92f26b4698b26 2023-04-15 02:55:54,920 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=49.20 KB at sequenceid=498 (bloomFilter=true), to=hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/a29fa882a9e3f50c391e3aa08c5c5e72/.tmp/C/c07ea2e71580443c9ae8193743672c05 2023-04-15 02:55:54,926 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/a29fa882a9e3f50c391e3aa08c5c5e72/.tmp/A/6ec73ecb341c44509a59650277eb89c1 as hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/a29fa882a9e3f50c391e3aa08c5c5e72/A/6ec73ecb341c44509a59650277eb89c1 2023-04-15 02:55:54,931 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/a29fa882a9e3f50c391e3aa08c5c5e72/A/6ec73ecb341c44509a59650277eb89c1, entries=150, sequenceid=498, filesize=11.9 K 2023-04-15 02:55:54,932 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/a29fa882a9e3f50c391e3aa08c5c5e72/.tmp/B/ef39994cb7ec48cd9fb92f26b4698b26 as hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/a29fa882a9e3f50c391e3aa08c5c5e72/B/ef39994cb7ec48cd9fb92f26b4698b26 2023-04-15 02:55:54,938 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/a29fa882a9e3f50c391e3aa08c5c5e72/B/ef39994cb7ec48cd9fb92f26b4698b26, entries=150, sequenceid=498, filesize=11.9 K 2023-04-15 02:55:54,939 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/a29fa882a9e3f50c391e3aa08c5c5e72/.tmp/C/c07ea2e71580443c9ae8193743672c05 as hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/a29fa882a9e3f50c391e3aa08c5c5e72/C/c07ea2e71580443c9ae8193743672c05 2023-04-15 02:55:54,944 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/a29fa882a9e3f50c391e3aa08c5c5e72/C/c07ea2e71580443c9ae8193743672c05, entries=150, sequenceid=498, filesize=11.9 K 2023-04-15 02:55:54,945 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~147.60 KB/151140, heapSize ~410.72 KB/420576, currentSize=26.84 KB/27480 for a29fa882a9e3f50c391e3aa08c5c5e72 in 872ms, sequenceid=498, compaction requested=false 2023-04-15 02:55:54,945 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for a29fa882a9e3f50c391e3aa08c5c5e72: 2023-04-15 02:55:54,945 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1681527331158.a29fa882a9e3f50c391e3aa08c5c5e72. 2023-04-15 02:55:54,945 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1681527331158.a29fa882a9e3f50c391e3aa08c5c5e72. 2023-04-15 02:55:54,945 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1681527331158.a29fa882a9e3f50c391e3aa08c5c5e72. after waiting 0 ms 2023-04-15 02:55:54,945 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1681527331158.a29fa882a9e3f50c391e3aa08c5c5e72. 2023-04-15 02:55:54,945 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(2745): Flushing a29fa882a9e3f50c391e3aa08c5c5e72 3/3 column families, dataSize=26.84 KB heapSize=80.44 KB 2023-04-15 02:55:54,945 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a29fa882a9e3f50c391e3aa08c5c5e72, store=A 2023-04-15 02:55:54,946 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-15 02:55:54,946 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a29fa882a9e3f50c391e3aa08c5c5e72, store=B 2023-04-15 02:55:54,946 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-15 02:55:54,946 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a29fa882a9e3f50c391e3aa08c5c5e72, store=C 2023-04-15 02:55:54,946 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-15 02:55:55,356 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=505 (bloomFilter=true), to=hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/a29fa882a9e3f50c391e3aa08c5c5e72/.tmp/A/8e57c5759618496b882a08e61652edbc 2023-04-15 02:55:55,388 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(1227): Checking to see if procedure is done pid=50 2023-04-15 02:55:55,771 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=505 (bloomFilter=true), to=hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/a29fa882a9e3f50c391e3aa08c5c5e72/.tmp/B/2c8a05257dd04ce7a656c414cd1c6fae 2023-04-15 02:55:55,787 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=505 (bloomFilter=true), to=hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/a29fa882a9e3f50c391e3aa08c5c5e72/.tmp/C/88d606c95a3e4d27b7727f29fa316b53 2023-04-15 02:55:55,792 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/a29fa882a9e3f50c391e3aa08c5c5e72/.tmp/A/8e57c5759618496b882a08e61652edbc as hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/a29fa882a9e3f50c391e3aa08c5c5e72/A/8e57c5759618496b882a08e61652edbc 2023-04-15 02:55:55,796 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/a29fa882a9e3f50c391e3aa08c5c5e72/A/8e57c5759618496b882a08e61652edbc, entries=150, sequenceid=505, filesize=11.9 K 2023-04-15 02:55:55,797 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/a29fa882a9e3f50c391e3aa08c5c5e72/.tmp/B/2c8a05257dd04ce7a656c414cd1c6fae as hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/a29fa882a9e3f50c391e3aa08c5c5e72/B/2c8a05257dd04ce7a656c414cd1c6fae 2023-04-15 02:55:55,800 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/a29fa882a9e3f50c391e3aa08c5c5e72/B/2c8a05257dd04ce7a656c414cd1c6fae, entries=150, sequenceid=505, filesize=11.9 K 2023-04-15 02:55:55,800 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/a29fa882a9e3f50c391e3aa08c5c5e72/.tmp/C/88d606c95a3e4d27b7727f29fa316b53 as hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/a29fa882a9e3f50c391e3aa08c5c5e72/C/88d606c95a3e4d27b7727f29fa316b53 2023-04-15 02:55:55,803 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/a29fa882a9e3f50c391e3aa08c5c5e72/C/88d606c95a3e4d27b7727f29fa316b53, entries=150, sequenceid=505, filesize=11.9 K 2023-04-15 02:55:55,804 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(2948): Finished flush of dataSize ~26.84 KB/27480, heapSize ~80.39 KB/82320, currentSize=0 B/0 for a29fa882a9e3f50c391e3aa08c5c5e72 in 859ms, sequenceid=505, compaction requested=true 2023-04-15 02:55:55,825 DEBUG [StoreCloser-TestAcidGuarantees,,1681527331158.a29fa882a9e3f50c391e3aa08c5c5e72.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/a29fa882a9e3f50c391e3aa08c5c5e72/A/0e96c61d77dc4e7196e63b48bf12904c, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/a29fa882a9e3f50c391e3aa08c5c5e72/A/8b1cd7b07fb14077863a5c54dada63bb, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/a29fa882a9e3f50c391e3aa08c5c5e72/A/4f1a4b7179234562828b589c7d2475e0, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/a29fa882a9e3f50c391e3aa08c5c5e72/A/ade7f87389884e4ea915a78f74f55f27, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/a29fa882a9e3f50c391e3aa08c5c5e72/A/bee8d060091749a4af47caf99098c5bb, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/a29fa882a9e3f50c391e3aa08c5c5e72/A/02ceb979dd0b4fde839b9ab0388624be, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/a29fa882a9e3f50c391e3aa08c5c5e72/A/b802fc417bb640f5bd6179acfd948549, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/a29fa882a9e3f50c391e3aa08c5c5e72/A/b0e063b9c503473580ad242c3fc542ea, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/a29fa882a9e3f50c391e3aa08c5c5e72/A/566f729b22db4659ab9e6ff55d4bfc41, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/a29fa882a9e3f50c391e3aa08c5c5e72/A/e968895af35f4ae7a46c8b0e22f49d25, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/a29fa882a9e3f50c391e3aa08c5c5e72/A/38b4f627839a4d0abd642ffe8d380237, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/a29fa882a9e3f50c391e3aa08c5c5e72/A/4ab24298f79e488abab5ebc46292e6b3, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/a29fa882a9e3f50c391e3aa08c5c5e72/A/d2705aa6585d42dba39e3bbfd4abf7ef, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/a29fa882a9e3f50c391e3aa08c5c5e72/A/1825e46b7b414112bf098f0a6f0c0c09, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/a29fa882a9e3f50c391e3aa08c5c5e72/A/bf1a5b878def417280c39297674887c5, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/a29fa882a9e3f50c391e3aa08c5c5e72/A/2791a854894341ac90b837693a2a8125, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/a29fa882a9e3f50c391e3aa08c5c5e72/A/6637bb1f67c841e3a1430bf06afb4102, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/a29fa882a9e3f50c391e3aa08c5c5e72/A/cdcb1334de5b4aceb8a27f5d37d7cc43, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/a29fa882a9e3f50c391e3aa08c5c5e72/A/1e6fdf63e00641cf974a6d51d0c9c42f, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/a29fa882a9e3f50c391e3aa08c5c5e72/A/cb1cebeccfd7464b84d6eb0a083c50c0, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/a29fa882a9e3f50c391e3aa08c5c5e72/A/1ee1d258316345379a04f4b9bd87e5ed, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/a29fa882a9e3f50c391e3aa08c5c5e72/A/e92ab01b17d04e6692817e2ec14e1214, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/a29fa882a9e3f50c391e3aa08c5c5e72/A/6b6ea55ba4564b2885f935edea24a8df, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/a29fa882a9e3f50c391e3aa08c5c5e72/A/8b168677d78840c7bc3dba0bfc7fa4cc, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/a29fa882a9e3f50c391e3aa08c5c5e72/A/90e4b2984b0b4171b9e59884e699b38b, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/a29fa882a9e3f50c391e3aa08c5c5e72/A/b9c42cf4ab0b40199b787473de2b3b3d, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/a29fa882a9e3f50c391e3aa08c5c5e72/A/69c91ab709194d56b46e176b3215d30a, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/a29fa882a9e3f50c391e3aa08c5c5e72/A/9a723c5189444482886f596aa7488555, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/a29fa882a9e3f50c391e3aa08c5c5e72/A/efcbca754cc14ecdb9f9a36e94daa51e] to archive 2023-04-15 02:55:55,825 DEBUG [StoreCloser-TestAcidGuarantees,,1681527331158.a29fa882a9e3f50c391e3aa08c5c5e72.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-15 02:55:55,827 DEBUG [StoreCloser-TestAcidGuarantees,,1681527331158.a29fa882a9e3f50c391e3aa08c5c5e72.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/a29fa882a9e3f50c391e3aa08c5c5e72/A/0e96c61d77dc4e7196e63b48bf12904c to hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/archive/data/default/TestAcidGuarantees/a29fa882a9e3f50c391e3aa08c5c5e72/A/0e96c61d77dc4e7196e63b48bf12904c 2023-04-15 02:55:55,827 DEBUG [StoreCloser-TestAcidGuarantees,,1681527331158.a29fa882a9e3f50c391e3aa08c5c5e72.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/a29fa882a9e3f50c391e3aa08c5c5e72/A/8b1cd7b07fb14077863a5c54dada63bb to hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/archive/data/default/TestAcidGuarantees/a29fa882a9e3f50c391e3aa08c5c5e72/A/8b1cd7b07fb14077863a5c54dada63bb 2023-04-15 02:55:55,828 DEBUG [StoreCloser-TestAcidGuarantees,,1681527331158.a29fa882a9e3f50c391e3aa08c5c5e72.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/a29fa882a9e3f50c391e3aa08c5c5e72/A/4f1a4b7179234562828b589c7d2475e0 to hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/archive/data/default/TestAcidGuarantees/a29fa882a9e3f50c391e3aa08c5c5e72/A/4f1a4b7179234562828b589c7d2475e0 2023-04-15 02:55:55,829 DEBUG [StoreCloser-TestAcidGuarantees,,1681527331158.a29fa882a9e3f50c391e3aa08c5c5e72.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/a29fa882a9e3f50c391e3aa08c5c5e72/A/ade7f87389884e4ea915a78f74f55f27 to hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/archive/data/default/TestAcidGuarantees/a29fa882a9e3f50c391e3aa08c5c5e72/A/ade7f87389884e4ea915a78f74f55f27 2023-04-15 02:55:55,830 DEBUG [StoreCloser-TestAcidGuarantees,,1681527331158.a29fa882a9e3f50c391e3aa08c5c5e72.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/a29fa882a9e3f50c391e3aa08c5c5e72/A/bee8d060091749a4af47caf99098c5bb to hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/archive/data/default/TestAcidGuarantees/a29fa882a9e3f50c391e3aa08c5c5e72/A/bee8d060091749a4af47caf99098c5bb 2023-04-15 02:55:55,831 DEBUG [StoreCloser-TestAcidGuarantees,,1681527331158.a29fa882a9e3f50c391e3aa08c5c5e72.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/a29fa882a9e3f50c391e3aa08c5c5e72/A/02ceb979dd0b4fde839b9ab0388624be to hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/archive/data/default/TestAcidGuarantees/a29fa882a9e3f50c391e3aa08c5c5e72/A/02ceb979dd0b4fde839b9ab0388624be 2023-04-15 02:55:55,832 DEBUG [StoreCloser-TestAcidGuarantees,,1681527331158.a29fa882a9e3f50c391e3aa08c5c5e72.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/a29fa882a9e3f50c391e3aa08c5c5e72/A/b802fc417bb640f5bd6179acfd948549 to hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/archive/data/default/TestAcidGuarantees/a29fa882a9e3f50c391e3aa08c5c5e72/A/b802fc417bb640f5bd6179acfd948549 2023-04-15 02:55:55,833 DEBUG [StoreCloser-TestAcidGuarantees,,1681527331158.a29fa882a9e3f50c391e3aa08c5c5e72.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/a29fa882a9e3f50c391e3aa08c5c5e72/A/b0e063b9c503473580ad242c3fc542ea to hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/archive/data/default/TestAcidGuarantees/a29fa882a9e3f50c391e3aa08c5c5e72/A/b0e063b9c503473580ad242c3fc542ea 2023-04-15 02:55:55,834 DEBUG [StoreCloser-TestAcidGuarantees,,1681527331158.a29fa882a9e3f50c391e3aa08c5c5e72.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/a29fa882a9e3f50c391e3aa08c5c5e72/A/566f729b22db4659ab9e6ff55d4bfc41 to hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/archive/data/default/TestAcidGuarantees/a29fa882a9e3f50c391e3aa08c5c5e72/A/566f729b22db4659ab9e6ff55d4bfc41 2023-04-15 02:55:55,835 DEBUG [StoreCloser-TestAcidGuarantees,,1681527331158.a29fa882a9e3f50c391e3aa08c5c5e72.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/a29fa882a9e3f50c391e3aa08c5c5e72/A/e968895af35f4ae7a46c8b0e22f49d25 to hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/archive/data/default/TestAcidGuarantees/a29fa882a9e3f50c391e3aa08c5c5e72/A/e968895af35f4ae7a46c8b0e22f49d25 2023-04-15 02:55:55,836 DEBUG [StoreCloser-TestAcidGuarantees,,1681527331158.a29fa882a9e3f50c391e3aa08c5c5e72.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/a29fa882a9e3f50c391e3aa08c5c5e72/A/38b4f627839a4d0abd642ffe8d380237 to hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/archive/data/default/TestAcidGuarantees/a29fa882a9e3f50c391e3aa08c5c5e72/A/38b4f627839a4d0abd642ffe8d380237 2023-04-15 02:55:55,837 DEBUG [StoreCloser-TestAcidGuarantees,,1681527331158.a29fa882a9e3f50c391e3aa08c5c5e72.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/a29fa882a9e3f50c391e3aa08c5c5e72/A/4ab24298f79e488abab5ebc46292e6b3 to hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/archive/data/default/TestAcidGuarantees/a29fa882a9e3f50c391e3aa08c5c5e72/A/4ab24298f79e488abab5ebc46292e6b3 2023-04-15 02:55:55,837 DEBUG [StoreCloser-TestAcidGuarantees,,1681527331158.a29fa882a9e3f50c391e3aa08c5c5e72.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/a29fa882a9e3f50c391e3aa08c5c5e72/A/d2705aa6585d42dba39e3bbfd4abf7ef to hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/archive/data/default/TestAcidGuarantees/a29fa882a9e3f50c391e3aa08c5c5e72/A/d2705aa6585d42dba39e3bbfd4abf7ef 2023-04-15 02:55:55,838 DEBUG [StoreCloser-TestAcidGuarantees,,1681527331158.a29fa882a9e3f50c391e3aa08c5c5e72.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/a29fa882a9e3f50c391e3aa08c5c5e72/A/1825e46b7b414112bf098f0a6f0c0c09 to hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/archive/data/default/TestAcidGuarantees/a29fa882a9e3f50c391e3aa08c5c5e72/A/1825e46b7b414112bf098f0a6f0c0c09 2023-04-15 02:55:55,839 DEBUG [StoreCloser-TestAcidGuarantees,,1681527331158.a29fa882a9e3f50c391e3aa08c5c5e72.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/a29fa882a9e3f50c391e3aa08c5c5e72/A/bf1a5b878def417280c39297674887c5 to hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/archive/data/default/TestAcidGuarantees/a29fa882a9e3f50c391e3aa08c5c5e72/A/bf1a5b878def417280c39297674887c5 2023-04-15 02:55:55,840 DEBUG [StoreCloser-TestAcidGuarantees,,1681527331158.a29fa882a9e3f50c391e3aa08c5c5e72.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/a29fa882a9e3f50c391e3aa08c5c5e72/A/2791a854894341ac90b837693a2a8125 to hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/archive/data/default/TestAcidGuarantees/a29fa882a9e3f50c391e3aa08c5c5e72/A/2791a854894341ac90b837693a2a8125 2023-04-15 02:55:55,841 DEBUG [StoreCloser-TestAcidGuarantees,,1681527331158.a29fa882a9e3f50c391e3aa08c5c5e72.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/a29fa882a9e3f50c391e3aa08c5c5e72/A/6637bb1f67c841e3a1430bf06afb4102 to hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/archive/data/default/TestAcidGuarantees/a29fa882a9e3f50c391e3aa08c5c5e72/A/6637bb1f67c841e3a1430bf06afb4102 2023-04-15 02:55:55,841 DEBUG [StoreCloser-TestAcidGuarantees,,1681527331158.a29fa882a9e3f50c391e3aa08c5c5e72.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/a29fa882a9e3f50c391e3aa08c5c5e72/A/cdcb1334de5b4aceb8a27f5d37d7cc43 to hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/archive/data/default/TestAcidGuarantees/a29fa882a9e3f50c391e3aa08c5c5e72/A/cdcb1334de5b4aceb8a27f5d37d7cc43 2023-04-15 02:55:55,842 DEBUG [StoreCloser-TestAcidGuarantees,,1681527331158.a29fa882a9e3f50c391e3aa08c5c5e72.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/a29fa882a9e3f50c391e3aa08c5c5e72/A/1e6fdf63e00641cf974a6d51d0c9c42f to hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/archive/data/default/TestAcidGuarantees/a29fa882a9e3f50c391e3aa08c5c5e72/A/1e6fdf63e00641cf974a6d51d0c9c42f 2023-04-15 02:55:55,843 DEBUG [StoreCloser-TestAcidGuarantees,,1681527331158.a29fa882a9e3f50c391e3aa08c5c5e72.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/a29fa882a9e3f50c391e3aa08c5c5e72/A/cb1cebeccfd7464b84d6eb0a083c50c0 to hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/archive/data/default/TestAcidGuarantees/a29fa882a9e3f50c391e3aa08c5c5e72/A/cb1cebeccfd7464b84d6eb0a083c50c0 2023-04-15 02:55:55,844 DEBUG [StoreCloser-TestAcidGuarantees,,1681527331158.a29fa882a9e3f50c391e3aa08c5c5e72.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/a29fa882a9e3f50c391e3aa08c5c5e72/A/1ee1d258316345379a04f4b9bd87e5ed to hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/archive/data/default/TestAcidGuarantees/a29fa882a9e3f50c391e3aa08c5c5e72/A/1ee1d258316345379a04f4b9bd87e5ed 2023-04-15 02:55:55,844 DEBUG [StoreCloser-TestAcidGuarantees,,1681527331158.a29fa882a9e3f50c391e3aa08c5c5e72.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/a29fa882a9e3f50c391e3aa08c5c5e72/A/e92ab01b17d04e6692817e2ec14e1214 to hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/archive/data/default/TestAcidGuarantees/a29fa882a9e3f50c391e3aa08c5c5e72/A/e92ab01b17d04e6692817e2ec14e1214 2023-04-15 02:55:55,845 DEBUG [StoreCloser-TestAcidGuarantees,,1681527331158.a29fa882a9e3f50c391e3aa08c5c5e72.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/a29fa882a9e3f50c391e3aa08c5c5e72/A/6b6ea55ba4564b2885f935edea24a8df to hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/archive/data/default/TestAcidGuarantees/a29fa882a9e3f50c391e3aa08c5c5e72/A/6b6ea55ba4564b2885f935edea24a8df 2023-04-15 02:55:55,846 DEBUG [StoreCloser-TestAcidGuarantees,,1681527331158.a29fa882a9e3f50c391e3aa08c5c5e72.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/a29fa882a9e3f50c391e3aa08c5c5e72/A/8b168677d78840c7bc3dba0bfc7fa4cc to hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/archive/data/default/TestAcidGuarantees/a29fa882a9e3f50c391e3aa08c5c5e72/A/8b168677d78840c7bc3dba0bfc7fa4cc 2023-04-15 02:55:55,847 DEBUG [StoreCloser-TestAcidGuarantees,,1681527331158.a29fa882a9e3f50c391e3aa08c5c5e72.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/a29fa882a9e3f50c391e3aa08c5c5e72/A/90e4b2984b0b4171b9e59884e699b38b to hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/archive/data/default/TestAcidGuarantees/a29fa882a9e3f50c391e3aa08c5c5e72/A/90e4b2984b0b4171b9e59884e699b38b 2023-04-15 02:55:55,848 DEBUG [StoreCloser-TestAcidGuarantees,,1681527331158.a29fa882a9e3f50c391e3aa08c5c5e72.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/a29fa882a9e3f50c391e3aa08c5c5e72/A/b9c42cf4ab0b40199b787473de2b3b3d to hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/archive/data/default/TestAcidGuarantees/a29fa882a9e3f50c391e3aa08c5c5e72/A/b9c42cf4ab0b40199b787473de2b3b3d 2023-04-15 02:55:55,848 DEBUG [StoreCloser-TestAcidGuarantees,,1681527331158.a29fa882a9e3f50c391e3aa08c5c5e72.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/a29fa882a9e3f50c391e3aa08c5c5e72/A/69c91ab709194d56b46e176b3215d30a to hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/archive/data/default/TestAcidGuarantees/a29fa882a9e3f50c391e3aa08c5c5e72/A/69c91ab709194d56b46e176b3215d30a 2023-04-15 02:55:55,849 DEBUG [StoreCloser-TestAcidGuarantees,,1681527331158.a29fa882a9e3f50c391e3aa08c5c5e72.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/a29fa882a9e3f50c391e3aa08c5c5e72/A/9a723c5189444482886f596aa7488555 to hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/archive/data/default/TestAcidGuarantees/a29fa882a9e3f50c391e3aa08c5c5e72/A/9a723c5189444482886f596aa7488555 2023-04-15 02:55:55,850 DEBUG [StoreCloser-TestAcidGuarantees,,1681527331158.a29fa882a9e3f50c391e3aa08c5c5e72.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/a29fa882a9e3f50c391e3aa08c5c5e72/A/efcbca754cc14ecdb9f9a36e94daa51e to hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/archive/data/default/TestAcidGuarantees/a29fa882a9e3f50c391e3aa08c5c5e72/A/efcbca754cc14ecdb9f9a36e94daa51e 2023-04-15 02:55:55,852 DEBUG [StoreCloser-TestAcidGuarantees,,1681527331158.a29fa882a9e3f50c391e3aa08c5c5e72.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/a29fa882a9e3f50c391e3aa08c5c5e72/B/7c8b6c7189e14c99b6df874cccccf3e4, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/a29fa882a9e3f50c391e3aa08c5c5e72/B/d17591d53587463eb818d66612ab4d0b, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/a29fa882a9e3f50c391e3aa08c5c5e72/B/e3ff06b400854006852039779a95aade, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/a29fa882a9e3f50c391e3aa08c5c5e72/B/043e5d05a3184d83a9424af96a48be54, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/a29fa882a9e3f50c391e3aa08c5c5e72/B/c05c1b5850724aa8b501d9b7acd62a0c, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/a29fa882a9e3f50c391e3aa08c5c5e72/B/2df1e3eb809d43c0855e5c6a84429a07, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/a29fa882a9e3f50c391e3aa08c5c5e72/B/20e1b005b5354aee8174fe0b7cac8965, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/a29fa882a9e3f50c391e3aa08c5c5e72/B/8669bced50064421bc35c2bbf8ba2f27, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/a29fa882a9e3f50c391e3aa08c5c5e72/B/6af0f1b08cef4b61889eb01e2a3de70b, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/a29fa882a9e3f50c391e3aa08c5c5e72/B/d66b99d635ea4fa0adb9e73f822396bf, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/a29fa882a9e3f50c391e3aa08c5c5e72/B/af8ce0b6d48546668e62dac87f2fbbb9, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/a29fa882a9e3f50c391e3aa08c5c5e72/B/c6afbb3f63ab49ae98fca6be4fd386a0, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/a29fa882a9e3f50c391e3aa08c5c5e72/B/28da064938ee4d62a7003f2600f11230, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/a29fa882a9e3f50c391e3aa08c5c5e72/B/0173ef12b1354efb842e1ce4695fc4cd, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/a29fa882a9e3f50c391e3aa08c5c5e72/B/7feccce09cba4773982cddef4fc1b5f3, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/a29fa882a9e3f50c391e3aa08c5c5e72/B/13cb677837cb482b924e6abbb42c6a41, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/a29fa882a9e3f50c391e3aa08c5c5e72/B/7876501289af46cfbf6bfb8b5d2d89f2, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/a29fa882a9e3f50c391e3aa08c5c5e72/B/6300d8f8bbcd462081165d7f3228ba43, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/a29fa882a9e3f50c391e3aa08c5c5e72/B/47c3b4716c284989b651f97044b9111d, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/a29fa882a9e3f50c391e3aa08c5c5e72/B/f6e80b7413c74242b9c7c04ee2582e18, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/a29fa882a9e3f50c391e3aa08c5c5e72/B/1f7949a7a669426ab732be2e19ad4f6d, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/a29fa882a9e3f50c391e3aa08c5c5e72/B/6ba49c87f56a4a3cb7b3c3b548430ce1, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/a29fa882a9e3f50c391e3aa08c5c5e72/B/558068e1a6fd477385b4c9b7a02a5be1, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/a29fa882a9e3f50c391e3aa08c5c5e72/B/4526dc8ea3d145818bce0115e18a3c96, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/a29fa882a9e3f50c391e3aa08c5c5e72/B/513f80e4f4a141dc97f9f4db67f57ba6, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/a29fa882a9e3f50c391e3aa08c5c5e72/B/728653d7917c45ad8e9b5fe14c5a1cd1, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/a29fa882a9e3f50c391e3aa08c5c5e72/B/5aa162be14cf46b997e28b721c0920aa, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/a29fa882a9e3f50c391e3aa08c5c5e72/B/53153d4147c2459c959a071addb308ab, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/a29fa882a9e3f50c391e3aa08c5c5e72/B/5540a2e518584131ba9305c4dbf6db1a] to archive 2023-04-15 02:55:55,853 DEBUG [StoreCloser-TestAcidGuarantees,,1681527331158.a29fa882a9e3f50c391e3aa08c5c5e72.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-15 02:55:55,854 DEBUG [StoreCloser-TestAcidGuarantees,,1681527331158.a29fa882a9e3f50c391e3aa08c5c5e72.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/a29fa882a9e3f50c391e3aa08c5c5e72/B/7c8b6c7189e14c99b6df874cccccf3e4 to hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/archive/data/default/TestAcidGuarantees/a29fa882a9e3f50c391e3aa08c5c5e72/B/7c8b6c7189e14c99b6df874cccccf3e4 2023-04-15 02:55:55,855 DEBUG [StoreCloser-TestAcidGuarantees,,1681527331158.a29fa882a9e3f50c391e3aa08c5c5e72.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/a29fa882a9e3f50c391e3aa08c5c5e72/B/d17591d53587463eb818d66612ab4d0b to hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/archive/data/default/TestAcidGuarantees/a29fa882a9e3f50c391e3aa08c5c5e72/B/d17591d53587463eb818d66612ab4d0b 2023-04-15 02:55:55,856 DEBUG [StoreCloser-TestAcidGuarantees,,1681527331158.a29fa882a9e3f50c391e3aa08c5c5e72.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/a29fa882a9e3f50c391e3aa08c5c5e72/B/e3ff06b400854006852039779a95aade to hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/archive/data/default/TestAcidGuarantees/a29fa882a9e3f50c391e3aa08c5c5e72/B/e3ff06b400854006852039779a95aade 2023-04-15 02:55:55,856 DEBUG [StoreCloser-TestAcidGuarantees,,1681527331158.a29fa882a9e3f50c391e3aa08c5c5e72.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/a29fa882a9e3f50c391e3aa08c5c5e72/B/043e5d05a3184d83a9424af96a48be54 to hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/archive/data/default/TestAcidGuarantees/a29fa882a9e3f50c391e3aa08c5c5e72/B/043e5d05a3184d83a9424af96a48be54 2023-04-15 02:55:55,857 DEBUG [StoreCloser-TestAcidGuarantees,,1681527331158.a29fa882a9e3f50c391e3aa08c5c5e72.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/a29fa882a9e3f50c391e3aa08c5c5e72/B/c05c1b5850724aa8b501d9b7acd62a0c to hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/archive/data/default/TestAcidGuarantees/a29fa882a9e3f50c391e3aa08c5c5e72/B/c05c1b5850724aa8b501d9b7acd62a0c 2023-04-15 02:55:55,858 DEBUG [StoreCloser-TestAcidGuarantees,,1681527331158.a29fa882a9e3f50c391e3aa08c5c5e72.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/a29fa882a9e3f50c391e3aa08c5c5e72/B/2df1e3eb809d43c0855e5c6a84429a07 to hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/archive/data/default/TestAcidGuarantees/a29fa882a9e3f50c391e3aa08c5c5e72/B/2df1e3eb809d43c0855e5c6a84429a07 2023-04-15 02:55:55,859 DEBUG [StoreCloser-TestAcidGuarantees,,1681527331158.a29fa882a9e3f50c391e3aa08c5c5e72.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/a29fa882a9e3f50c391e3aa08c5c5e72/B/20e1b005b5354aee8174fe0b7cac8965 to hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/archive/data/default/TestAcidGuarantees/a29fa882a9e3f50c391e3aa08c5c5e72/B/20e1b005b5354aee8174fe0b7cac8965 2023-04-15 02:55:55,860 DEBUG [StoreCloser-TestAcidGuarantees,,1681527331158.a29fa882a9e3f50c391e3aa08c5c5e72.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/a29fa882a9e3f50c391e3aa08c5c5e72/B/8669bced50064421bc35c2bbf8ba2f27 to hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/archive/data/default/TestAcidGuarantees/a29fa882a9e3f50c391e3aa08c5c5e72/B/8669bced50064421bc35c2bbf8ba2f27 2023-04-15 02:55:55,861 DEBUG [StoreCloser-TestAcidGuarantees,,1681527331158.a29fa882a9e3f50c391e3aa08c5c5e72.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/a29fa882a9e3f50c391e3aa08c5c5e72/B/6af0f1b08cef4b61889eb01e2a3de70b to hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/archive/data/default/TestAcidGuarantees/a29fa882a9e3f50c391e3aa08c5c5e72/B/6af0f1b08cef4b61889eb01e2a3de70b 2023-04-15 02:55:55,861 DEBUG [StoreCloser-TestAcidGuarantees,,1681527331158.a29fa882a9e3f50c391e3aa08c5c5e72.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/a29fa882a9e3f50c391e3aa08c5c5e72/B/d66b99d635ea4fa0adb9e73f822396bf to hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/archive/data/default/TestAcidGuarantees/a29fa882a9e3f50c391e3aa08c5c5e72/B/d66b99d635ea4fa0adb9e73f822396bf 2023-04-15 02:55:55,862 DEBUG [StoreCloser-TestAcidGuarantees,,1681527331158.a29fa882a9e3f50c391e3aa08c5c5e72.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/a29fa882a9e3f50c391e3aa08c5c5e72/B/af8ce0b6d48546668e62dac87f2fbbb9 to hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/archive/data/default/TestAcidGuarantees/a29fa882a9e3f50c391e3aa08c5c5e72/B/af8ce0b6d48546668e62dac87f2fbbb9 2023-04-15 02:55:55,863 DEBUG [StoreCloser-TestAcidGuarantees,,1681527331158.a29fa882a9e3f50c391e3aa08c5c5e72.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/a29fa882a9e3f50c391e3aa08c5c5e72/B/c6afbb3f63ab49ae98fca6be4fd386a0 to hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/archive/data/default/TestAcidGuarantees/a29fa882a9e3f50c391e3aa08c5c5e72/B/c6afbb3f63ab49ae98fca6be4fd386a0 2023-04-15 02:55:55,864 DEBUG [StoreCloser-TestAcidGuarantees,,1681527331158.a29fa882a9e3f50c391e3aa08c5c5e72.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/a29fa882a9e3f50c391e3aa08c5c5e72/B/28da064938ee4d62a7003f2600f11230 to hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/archive/data/default/TestAcidGuarantees/a29fa882a9e3f50c391e3aa08c5c5e72/B/28da064938ee4d62a7003f2600f11230 2023-04-15 02:55:55,865 DEBUG [StoreCloser-TestAcidGuarantees,,1681527331158.a29fa882a9e3f50c391e3aa08c5c5e72.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/a29fa882a9e3f50c391e3aa08c5c5e72/B/0173ef12b1354efb842e1ce4695fc4cd to hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/archive/data/default/TestAcidGuarantees/a29fa882a9e3f50c391e3aa08c5c5e72/B/0173ef12b1354efb842e1ce4695fc4cd 2023-04-15 02:55:55,866 DEBUG [StoreCloser-TestAcidGuarantees,,1681527331158.a29fa882a9e3f50c391e3aa08c5c5e72.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/a29fa882a9e3f50c391e3aa08c5c5e72/B/7feccce09cba4773982cddef4fc1b5f3 to hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/archive/data/default/TestAcidGuarantees/a29fa882a9e3f50c391e3aa08c5c5e72/B/7feccce09cba4773982cddef4fc1b5f3 2023-04-15 02:55:55,867 DEBUG [StoreCloser-TestAcidGuarantees,,1681527331158.a29fa882a9e3f50c391e3aa08c5c5e72.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/a29fa882a9e3f50c391e3aa08c5c5e72/B/13cb677837cb482b924e6abbb42c6a41 to hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/archive/data/default/TestAcidGuarantees/a29fa882a9e3f50c391e3aa08c5c5e72/B/13cb677837cb482b924e6abbb42c6a41 2023-04-15 02:55:55,868 DEBUG [StoreCloser-TestAcidGuarantees,,1681527331158.a29fa882a9e3f50c391e3aa08c5c5e72.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/a29fa882a9e3f50c391e3aa08c5c5e72/B/7876501289af46cfbf6bfb8b5d2d89f2 to hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/archive/data/default/TestAcidGuarantees/a29fa882a9e3f50c391e3aa08c5c5e72/B/7876501289af46cfbf6bfb8b5d2d89f2 2023-04-15 02:55:55,869 DEBUG [StoreCloser-TestAcidGuarantees,,1681527331158.a29fa882a9e3f50c391e3aa08c5c5e72.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/a29fa882a9e3f50c391e3aa08c5c5e72/B/6300d8f8bbcd462081165d7f3228ba43 to hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/archive/data/default/TestAcidGuarantees/a29fa882a9e3f50c391e3aa08c5c5e72/B/6300d8f8bbcd462081165d7f3228ba43 2023-04-15 02:55:55,870 DEBUG [StoreCloser-TestAcidGuarantees,,1681527331158.a29fa882a9e3f50c391e3aa08c5c5e72.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/a29fa882a9e3f50c391e3aa08c5c5e72/B/47c3b4716c284989b651f97044b9111d to hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/archive/data/default/TestAcidGuarantees/a29fa882a9e3f50c391e3aa08c5c5e72/B/47c3b4716c284989b651f97044b9111d 2023-04-15 02:55:55,871 DEBUG [StoreCloser-TestAcidGuarantees,,1681527331158.a29fa882a9e3f50c391e3aa08c5c5e72.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/a29fa882a9e3f50c391e3aa08c5c5e72/B/f6e80b7413c74242b9c7c04ee2582e18 to hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/archive/data/default/TestAcidGuarantees/a29fa882a9e3f50c391e3aa08c5c5e72/B/f6e80b7413c74242b9c7c04ee2582e18 2023-04-15 02:55:55,872 DEBUG [StoreCloser-TestAcidGuarantees,,1681527331158.a29fa882a9e3f50c391e3aa08c5c5e72.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/a29fa882a9e3f50c391e3aa08c5c5e72/B/1f7949a7a669426ab732be2e19ad4f6d to hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/archive/data/default/TestAcidGuarantees/a29fa882a9e3f50c391e3aa08c5c5e72/B/1f7949a7a669426ab732be2e19ad4f6d 2023-04-15 02:55:55,873 DEBUG [StoreCloser-TestAcidGuarantees,,1681527331158.a29fa882a9e3f50c391e3aa08c5c5e72.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/a29fa882a9e3f50c391e3aa08c5c5e72/B/6ba49c87f56a4a3cb7b3c3b548430ce1 to hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/archive/data/default/TestAcidGuarantees/a29fa882a9e3f50c391e3aa08c5c5e72/B/6ba49c87f56a4a3cb7b3c3b548430ce1 2023-04-15 02:55:55,874 DEBUG [StoreCloser-TestAcidGuarantees,,1681527331158.a29fa882a9e3f50c391e3aa08c5c5e72.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/a29fa882a9e3f50c391e3aa08c5c5e72/B/558068e1a6fd477385b4c9b7a02a5be1 to hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/archive/data/default/TestAcidGuarantees/a29fa882a9e3f50c391e3aa08c5c5e72/B/558068e1a6fd477385b4c9b7a02a5be1 2023-04-15 02:55:55,875 DEBUG [StoreCloser-TestAcidGuarantees,,1681527331158.a29fa882a9e3f50c391e3aa08c5c5e72.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/a29fa882a9e3f50c391e3aa08c5c5e72/B/4526dc8ea3d145818bce0115e18a3c96 to hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/archive/data/default/TestAcidGuarantees/a29fa882a9e3f50c391e3aa08c5c5e72/B/4526dc8ea3d145818bce0115e18a3c96 2023-04-15 02:55:55,876 DEBUG [StoreCloser-TestAcidGuarantees,,1681527331158.a29fa882a9e3f50c391e3aa08c5c5e72.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/a29fa882a9e3f50c391e3aa08c5c5e72/B/513f80e4f4a141dc97f9f4db67f57ba6 to hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/archive/data/default/TestAcidGuarantees/a29fa882a9e3f50c391e3aa08c5c5e72/B/513f80e4f4a141dc97f9f4db67f57ba6 2023-04-15 02:55:55,877 DEBUG [StoreCloser-TestAcidGuarantees,,1681527331158.a29fa882a9e3f50c391e3aa08c5c5e72.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/a29fa882a9e3f50c391e3aa08c5c5e72/B/728653d7917c45ad8e9b5fe14c5a1cd1 to hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/archive/data/default/TestAcidGuarantees/a29fa882a9e3f50c391e3aa08c5c5e72/B/728653d7917c45ad8e9b5fe14c5a1cd1 2023-04-15 02:55:55,879 DEBUG [StoreCloser-TestAcidGuarantees,,1681527331158.a29fa882a9e3f50c391e3aa08c5c5e72.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/a29fa882a9e3f50c391e3aa08c5c5e72/B/5aa162be14cf46b997e28b721c0920aa to hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/archive/data/default/TestAcidGuarantees/a29fa882a9e3f50c391e3aa08c5c5e72/B/5aa162be14cf46b997e28b721c0920aa 2023-04-15 02:55:55,880 DEBUG [StoreCloser-TestAcidGuarantees,,1681527331158.a29fa882a9e3f50c391e3aa08c5c5e72.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/a29fa882a9e3f50c391e3aa08c5c5e72/B/53153d4147c2459c959a071addb308ab to hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/archive/data/default/TestAcidGuarantees/a29fa882a9e3f50c391e3aa08c5c5e72/B/53153d4147c2459c959a071addb308ab 2023-04-15 02:55:55,881 DEBUG [StoreCloser-TestAcidGuarantees,,1681527331158.a29fa882a9e3f50c391e3aa08c5c5e72.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/a29fa882a9e3f50c391e3aa08c5c5e72/B/5540a2e518584131ba9305c4dbf6db1a to hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/archive/data/default/TestAcidGuarantees/a29fa882a9e3f50c391e3aa08c5c5e72/B/5540a2e518584131ba9305c4dbf6db1a 2023-04-15 02:55:55,885 DEBUG [StoreCloser-TestAcidGuarantees,,1681527331158.a29fa882a9e3f50c391e3aa08c5c5e72.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/a29fa882a9e3f50c391e3aa08c5c5e72/C/4b027f62b2504671b04f87e52c4b6d59, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/a29fa882a9e3f50c391e3aa08c5c5e72/C/0a83cb0c7e0f46f6932c2f706cf22c5f, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/a29fa882a9e3f50c391e3aa08c5c5e72/C/451fc6fe095f4e2f8ac779de5c121c27, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/a29fa882a9e3f50c391e3aa08c5c5e72/C/a9f1e71f1b484838afc01f678f7f2456, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/a29fa882a9e3f50c391e3aa08c5c5e72/C/a55c129a820d464a8dd3cf851be769e5, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/a29fa882a9e3f50c391e3aa08c5c5e72/C/10f17357ba8c428a8595c4e8b9454a50, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/a29fa882a9e3f50c391e3aa08c5c5e72/C/c6ed2ac20dd844d894c05929f3482006, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/a29fa882a9e3f50c391e3aa08c5c5e72/C/7fdf48a747e543e1bc9b832166cde47a, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/a29fa882a9e3f50c391e3aa08c5c5e72/C/b871420ae8d8488395306a0bb12057fb, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/a29fa882a9e3f50c391e3aa08c5c5e72/C/7a21071acecd4947a6965346e0be8176, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/a29fa882a9e3f50c391e3aa08c5c5e72/C/ad509ed0bc7e4ae5b4d80a55856ce60f, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/a29fa882a9e3f50c391e3aa08c5c5e72/C/9499e2ae026b4cf7933d82cabe340d82, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/a29fa882a9e3f50c391e3aa08c5c5e72/C/8707dafd31ff48eb9c84d4e178e25eda, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/a29fa882a9e3f50c391e3aa08c5c5e72/C/258d4421a1fe497f89a4071b7edd4f2a, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/a29fa882a9e3f50c391e3aa08c5c5e72/C/1290cf8f5e884c469991519a85d516b9, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/a29fa882a9e3f50c391e3aa08c5c5e72/C/f7bec9a155c544618f64cf938f34ced0, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/a29fa882a9e3f50c391e3aa08c5c5e72/C/2a2de79b0f804d0aa7c763bfa19f47b1, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/a29fa882a9e3f50c391e3aa08c5c5e72/C/e0509719941d4e018f982030278d5c8c, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/a29fa882a9e3f50c391e3aa08c5c5e72/C/8e9b2ace06874a618b994b9577a16160, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/a29fa882a9e3f50c391e3aa08c5c5e72/C/0bb7d65eea50491994aff94d3a8396a5, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/a29fa882a9e3f50c391e3aa08c5c5e72/C/14944fc722fc45eba9b007613666b434, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/a29fa882a9e3f50c391e3aa08c5c5e72/C/49455a4d121741a59906287bbd6150b2, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/a29fa882a9e3f50c391e3aa08c5c5e72/C/35cbbdf3250541bb914fe4d9c46f472c, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/a29fa882a9e3f50c391e3aa08c5c5e72/C/a7ff1aaa2c07441988a294114bfbc8ee, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/a29fa882a9e3f50c391e3aa08c5c5e72/C/35d587b695f74c159d812876bf3420cc, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/a29fa882a9e3f50c391e3aa08c5c5e72/C/44bb03c191ed4eb5ae0c07767164cc4e, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/a29fa882a9e3f50c391e3aa08c5c5e72/C/d69e5dbe9ea14e0c876d82eb7d904f65, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/a29fa882a9e3f50c391e3aa08c5c5e72/C/b672c6f4c74140069d562b1d2bf96131, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/a29fa882a9e3f50c391e3aa08c5c5e72/C/2334d8d638eb47ddb44efe91c62993b2] to archive 2023-04-15 02:55:55,886 DEBUG [StoreCloser-TestAcidGuarantees,,1681527331158.a29fa882a9e3f50c391e3aa08c5c5e72.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-15 02:55:55,887 DEBUG [StoreCloser-TestAcidGuarantees,,1681527331158.a29fa882a9e3f50c391e3aa08c5c5e72.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/a29fa882a9e3f50c391e3aa08c5c5e72/C/4b027f62b2504671b04f87e52c4b6d59 to hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/archive/data/default/TestAcidGuarantees/a29fa882a9e3f50c391e3aa08c5c5e72/C/4b027f62b2504671b04f87e52c4b6d59 2023-04-15 02:55:55,888 DEBUG [StoreCloser-TestAcidGuarantees,,1681527331158.a29fa882a9e3f50c391e3aa08c5c5e72.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/a29fa882a9e3f50c391e3aa08c5c5e72/C/0a83cb0c7e0f46f6932c2f706cf22c5f to hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/archive/data/default/TestAcidGuarantees/a29fa882a9e3f50c391e3aa08c5c5e72/C/0a83cb0c7e0f46f6932c2f706cf22c5f 2023-04-15 02:55:55,889 DEBUG [StoreCloser-TestAcidGuarantees,,1681527331158.a29fa882a9e3f50c391e3aa08c5c5e72.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/a29fa882a9e3f50c391e3aa08c5c5e72/C/451fc6fe095f4e2f8ac779de5c121c27 to hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/archive/data/default/TestAcidGuarantees/a29fa882a9e3f50c391e3aa08c5c5e72/C/451fc6fe095f4e2f8ac779de5c121c27 2023-04-15 02:55:55,891 DEBUG [StoreCloser-TestAcidGuarantees,,1681527331158.a29fa882a9e3f50c391e3aa08c5c5e72.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/a29fa882a9e3f50c391e3aa08c5c5e72/C/a9f1e71f1b484838afc01f678f7f2456 to hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/archive/data/default/TestAcidGuarantees/a29fa882a9e3f50c391e3aa08c5c5e72/C/a9f1e71f1b484838afc01f678f7f2456 2023-04-15 02:55:55,892 DEBUG [StoreCloser-TestAcidGuarantees,,1681527331158.a29fa882a9e3f50c391e3aa08c5c5e72.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/a29fa882a9e3f50c391e3aa08c5c5e72/C/a55c129a820d464a8dd3cf851be769e5 to hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/archive/data/default/TestAcidGuarantees/a29fa882a9e3f50c391e3aa08c5c5e72/C/a55c129a820d464a8dd3cf851be769e5 2023-04-15 02:55:55,893 DEBUG [StoreCloser-TestAcidGuarantees,,1681527331158.a29fa882a9e3f50c391e3aa08c5c5e72.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/a29fa882a9e3f50c391e3aa08c5c5e72/C/10f17357ba8c428a8595c4e8b9454a50 to hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/archive/data/default/TestAcidGuarantees/a29fa882a9e3f50c391e3aa08c5c5e72/C/10f17357ba8c428a8595c4e8b9454a50 2023-04-15 02:55:55,894 DEBUG [StoreCloser-TestAcidGuarantees,,1681527331158.a29fa882a9e3f50c391e3aa08c5c5e72.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/a29fa882a9e3f50c391e3aa08c5c5e72/C/c6ed2ac20dd844d894c05929f3482006 to hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/archive/data/default/TestAcidGuarantees/a29fa882a9e3f50c391e3aa08c5c5e72/C/c6ed2ac20dd844d894c05929f3482006 2023-04-15 02:55:55,895 DEBUG [StoreCloser-TestAcidGuarantees,,1681527331158.a29fa882a9e3f50c391e3aa08c5c5e72.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/a29fa882a9e3f50c391e3aa08c5c5e72/C/7fdf48a747e543e1bc9b832166cde47a to hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/archive/data/default/TestAcidGuarantees/a29fa882a9e3f50c391e3aa08c5c5e72/C/7fdf48a747e543e1bc9b832166cde47a 2023-04-15 02:55:55,896 DEBUG [StoreCloser-TestAcidGuarantees,,1681527331158.a29fa882a9e3f50c391e3aa08c5c5e72.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/a29fa882a9e3f50c391e3aa08c5c5e72/C/b871420ae8d8488395306a0bb12057fb to hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/archive/data/default/TestAcidGuarantees/a29fa882a9e3f50c391e3aa08c5c5e72/C/b871420ae8d8488395306a0bb12057fb 2023-04-15 02:55:55,897 DEBUG [StoreCloser-TestAcidGuarantees,,1681527331158.a29fa882a9e3f50c391e3aa08c5c5e72.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/a29fa882a9e3f50c391e3aa08c5c5e72/C/7a21071acecd4947a6965346e0be8176 to hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/archive/data/default/TestAcidGuarantees/a29fa882a9e3f50c391e3aa08c5c5e72/C/7a21071acecd4947a6965346e0be8176 2023-04-15 02:55:55,898 DEBUG [StoreCloser-TestAcidGuarantees,,1681527331158.a29fa882a9e3f50c391e3aa08c5c5e72.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/a29fa882a9e3f50c391e3aa08c5c5e72/C/ad509ed0bc7e4ae5b4d80a55856ce60f to hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/archive/data/default/TestAcidGuarantees/a29fa882a9e3f50c391e3aa08c5c5e72/C/ad509ed0bc7e4ae5b4d80a55856ce60f 2023-04-15 02:55:55,899 DEBUG [StoreCloser-TestAcidGuarantees,,1681527331158.a29fa882a9e3f50c391e3aa08c5c5e72.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/a29fa882a9e3f50c391e3aa08c5c5e72/C/9499e2ae026b4cf7933d82cabe340d82 to hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/archive/data/default/TestAcidGuarantees/a29fa882a9e3f50c391e3aa08c5c5e72/C/9499e2ae026b4cf7933d82cabe340d82 2023-04-15 02:55:55,900 DEBUG [StoreCloser-TestAcidGuarantees,,1681527331158.a29fa882a9e3f50c391e3aa08c5c5e72.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/a29fa882a9e3f50c391e3aa08c5c5e72/C/8707dafd31ff48eb9c84d4e178e25eda to hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/archive/data/default/TestAcidGuarantees/a29fa882a9e3f50c391e3aa08c5c5e72/C/8707dafd31ff48eb9c84d4e178e25eda 2023-04-15 02:55:55,901 DEBUG [StoreCloser-TestAcidGuarantees,,1681527331158.a29fa882a9e3f50c391e3aa08c5c5e72.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/a29fa882a9e3f50c391e3aa08c5c5e72/C/258d4421a1fe497f89a4071b7edd4f2a to hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/archive/data/default/TestAcidGuarantees/a29fa882a9e3f50c391e3aa08c5c5e72/C/258d4421a1fe497f89a4071b7edd4f2a 2023-04-15 02:55:55,903 DEBUG [StoreCloser-TestAcidGuarantees,,1681527331158.a29fa882a9e3f50c391e3aa08c5c5e72.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/a29fa882a9e3f50c391e3aa08c5c5e72/C/1290cf8f5e884c469991519a85d516b9 to hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/archive/data/default/TestAcidGuarantees/a29fa882a9e3f50c391e3aa08c5c5e72/C/1290cf8f5e884c469991519a85d516b9 2023-04-15 02:55:55,904 DEBUG [StoreCloser-TestAcidGuarantees,,1681527331158.a29fa882a9e3f50c391e3aa08c5c5e72.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/a29fa882a9e3f50c391e3aa08c5c5e72/C/f7bec9a155c544618f64cf938f34ced0 to hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/archive/data/default/TestAcidGuarantees/a29fa882a9e3f50c391e3aa08c5c5e72/C/f7bec9a155c544618f64cf938f34ced0 2023-04-15 02:55:55,905 DEBUG [StoreCloser-TestAcidGuarantees,,1681527331158.a29fa882a9e3f50c391e3aa08c5c5e72.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/a29fa882a9e3f50c391e3aa08c5c5e72/C/2a2de79b0f804d0aa7c763bfa19f47b1 to hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/archive/data/default/TestAcidGuarantees/a29fa882a9e3f50c391e3aa08c5c5e72/C/2a2de79b0f804d0aa7c763bfa19f47b1 2023-04-15 02:55:55,906 DEBUG [StoreCloser-TestAcidGuarantees,,1681527331158.a29fa882a9e3f50c391e3aa08c5c5e72.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/a29fa882a9e3f50c391e3aa08c5c5e72/C/e0509719941d4e018f982030278d5c8c to hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/archive/data/default/TestAcidGuarantees/a29fa882a9e3f50c391e3aa08c5c5e72/C/e0509719941d4e018f982030278d5c8c 2023-04-15 02:55:55,908 DEBUG [StoreCloser-TestAcidGuarantees,,1681527331158.a29fa882a9e3f50c391e3aa08c5c5e72.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/a29fa882a9e3f50c391e3aa08c5c5e72/C/8e9b2ace06874a618b994b9577a16160 to hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/archive/data/default/TestAcidGuarantees/a29fa882a9e3f50c391e3aa08c5c5e72/C/8e9b2ace06874a618b994b9577a16160 2023-04-15 02:55:55,909 DEBUG [StoreCloser-TestAcidGuarantees,,1681527331158.a29fa882a9e3f50c391e3aa08c5c5e72.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/a29fa882a9e3f50c391e3aa08c5c5e72/C/0bb7d65eea50491994aff94d3a8396a5 to hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/archive/data/default/TestAcidGuarantees/a29fa882a9e3f50c391e3aa08c5c5e72/C/0bb7d65eea50491994aff94d3a8396a5 2023-04-15 02:55:55,910 DEBUG [StoreCloser-TestAcidGuarantees,,1681527331158.a29fa882a9e3f50c391e3aa08c5c5e72.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/a29fa882a9e3f50c391e3aa08c5c5e72/C/14944fc722fc45eba9b007613666b434 to hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/archive/data/default/TestAcidGuarantees/a29fa882a9e3f50c391e3aa08c5c5e72/C/14944fc722fc45eba9b007613666b434 2023-04-15 02:55:55,911 DEBUG [StoreCloser-TestAcidGuarantees,,1681527331158.a29fa882a9e3f50c391e3aa08c5c5e72.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/a29fa882a9e3f50c391e3aa08c5c5e72/C/49455a4d121741a59906287bbd6150b2 to hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/archive/data/default/TestAcidGuarantees/a29fa882a9e3f50c391e3aa08c5c5e72/C/49455a4d121741a59906287bbd6150b2 2023-04-15 02:55:55,912 DEBUG [StoreCloser-TestAcidGuarantees,,1681527331158.a29fa882a9e3f50c391e3aa08c5c5e72.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/a29fa882a9e3f50c391e3aa08c5c5e72/C/35cbbdf3250541bb914fe4d9c46f472c to hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/archive/data/default/TestAcidGuarantees/a29fa882a9e3f50c391e3aa08c5c5e72/C/35cbbdf3250541bb914fe4d9c46f472c 2023-04-15 02:55:55,913 DEBUG [StoreCloser-TestAcidGuarantees,,1681527331158.a29fa882a9e3f50c391e3aa08c5c5e72.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/a29fa882a9e3f50c391e3aa08c5c5e72/C/a7ff1aaa2c07441988a294114bfbc8ee to hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/archive/data/default/TestAcidGuarantees/a29fa882a9e3f50c391e3aa08c5c5e72/C/a7ff1aaa2c07441988a294114bfbc8ee 2023-04-15 02:55:55,914 DEBUG [StoreCloser-TestAcidGuarantees,,1681527331158.a29fa882a9e3f50c391e3aa08c5c5e72.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/a29fa882a9e3f50c391e3aa08c5c5e72/C/35d587b695f74c159d812876bf3420cc to hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/archive/data/default/TestAcidGuarantees/a29fa882a9e3f50c391e3aa08c5c5e72/C/35d587b695f74c159d812876bf3420cc 2023-04-15 02:55:55,915 DEBUG [StoreCloser-TestAcidGuarantees,,1681527331158.a29fa882a9e3f50c391e3aa08c5c5e72.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/a29fa882a9e3f50c391e3aa08c5c5e72/C/44bb03c191ed4eb5ae0c07767164cc4e to hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/archive/data/default/TestAcidGuarantees/a29fa882a9e3f50c391e3aa08c5c5e72/C/44bb03c191ed4eb5ae0c07767164cc4e 2023-04-15 02:55:55,916 DEBUG [StoreCloser-TestAcidGuarantees,,1681527331158.a29fa882a9e3f50c391e3aa08c5c5e72.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/a29fa882a9e3f50c391e3aa08c5c5e72/C/d69e5dbe9ea14e0c876d82eb7d904f65 to hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/archive/data/default/TestAcidGuarantees/a29fa882a9e3f50c391e3aa08c5c5e72/C/d69e5dbe9ea14e0c876d82eb7d904f65 2023-04-15 02:55:55,916 DEBUG [StoreCloser-TestAcidGuarantees,,1681527331158.a29fa882a9e3f50c391e3aa08c5c5e72.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/a29fa882a9e3f50c391e3aa08c5c5e72/C/b672c6f4c74140069d562b1d2bf96131 to hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/archive/data/default/TestAcidGuarantees/a29fa882a9e3f50c391e3aa08c5c5e72/C/b672c6f4c74140069d562b1d2bf96131 2023-04-15 02:55:55,917 DEBUG [StoreCloser-TestAcidGuarantees,,1681527331158.a29fa882a9e3f50c391e3aa08c5c5e72.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/a29fa882a9e3f50c391e3aa08c5c5e72/C/2334d8d638eb47ddb44efe91c62993b2 to hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/archive/data/default/TestAcidGuarantees/a29fa882a9e3f50c391e3aa08c5c5e72/C/2334d8d638eb47ddb44efe91c62993b2 2023-04-15 02:55:55,920 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/a29fa882a9e3f50c391e3aa08c5c5e72/recovered.edits/508.seqid, newMaxSeqId=508, maxSeqId=1 2023-04-15 02:55:55,922 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1681527331158.a29fa882a9e3f50c391e3aa08c5c5e72. 2023-04-15 02:55:55,922 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1558): Region close journal for a29fa882a9e3f50c391e3aa08c5c5e72: 2023-04-15 02:55:55,923 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] handler.UnassignRegionHandler(147): Closed a29fa882a9e3f50c391e3aa08c5c5e72 2023-04-15 02:55:55,923 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=51 updating hbase:meta row=a29fa882a9e3f50c391e3aa08c5c5e72, regionState=CLOSED 2023-04-15 02:55:55,924 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1681527331158.a29fa882a9e3f50c391e3aa08c5c5e72.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1681527355923"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1681527355923"}]},"ts":"1681527355923"} 2023-04-15 02:55:55,926 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=52, resume processing ppid=51 2023-04-15 02:55:55,926 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=52, ppid=51, state=SUCCESS; CloseRegionProcedure a29fa882a9e3f50c391e3aa08c5c5e72, server=jenkins-hbase20.apache.org,34681,1681527209133 in 1.6340 sec 2023-04-15 02:55:55,926 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=51, resume processing ppid=50 2023-04-15 02:55:55,927 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=51, ppid=50, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=a29fa882a9e3f50c391e3aa08c5c5e72, UNASSIGN in 1.6380 sec 2023-04-15 02:55:55,927 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1681527355927"}]},"ts":"1681527355927"} 2023-04-15 02:55:55,928 INFO [PEWorker-3] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=DISABLED in hbase:meta 2023-04-15 02:55:55,929 INFO [PEWorker-3] procedure.DisableTableProcedure(305): Set TestAcidGuarantees to state=DISABLED 2023-04-15 02:55:55,929 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=50, state=SUCCESS; DisableTableProcedure table=TestAcidGuarantees in 1.6540 sec 2023-04-15 02:55:56,389 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(1227): Checking to see if procedure is done pid=50 2023-04-15 02:55:56,389 INFO [Listener at localhost.localdomain/33693] client.HBaseAdmin$TableFuture(3541): Operation: DISABLE, Table Name: default:TestAcidGuarantees, procId: 50 completed 2023-04-15 02:55:56,390 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.HMaster$5(2228): Client=jenkins//148.251.75.209 delete TestAcidGuarantees 2023-04-15 02:55:56,391 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] procedure2.ProcedureExecutor(1029): Stored pid=53, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION; DeleteTableProcedure table=TestAcidGuarantees 2023-04-15 02:55:56,393 DEBUG [PEWorker-2] procedure.DeleteTableProcedure(101): Waiting for RIT for pid=53, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-15 02:55:56,394 DEBUG [PEWorker-2] procedure.DeleteTableProcedure(113): Deleting regions from filesystem for pid=53, state=RUNNABLE:DELETE_TABLE_CLEAR_FS_LAYOUT, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-15 02:55:56,394 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(1227): Checking to see if procedure is done pid=53 2023-04-15 02:55:56,401 DEBUG [HFileArchiver-11] backup.HFileArchiver(131): ARCHIVING hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/.tmp/data/default/TestAcidGuarantees/a29fa882a9e3f50c391e3aa08c5c5e72 2023-04-15 02:55:56,405 DEBUG [HFileArchiver-11] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/.tmp/data/default/TestAcidGuarantees/a29fa882a9e3f50c391e3aa08c5c5e72/A, FileablePath, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/.tmp/data/default/TestAcidGuarantees/a29fa882a9e3f50c391e3aa08c5c5e72/B, FileablePath, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/.tmp/data/default/TestAcidGuarantees/a29fa882a9e3f50c391e3aa08c5c5e72/C, FileablePath, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/.tmp/data/default/TestAcidGuarantees/a29fa882a9e3f50c391e3aa08c5c5e72/recovered.edits] 2023-04-15 02:55:56,409 DEBUG [HFileArchiver-11] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/.tmp/data/default/TestAcidGuarantees/a29fa882a9e3f50c391e3aa08c5c5e72/A/454801a3e8174a35b153f39efae8f0cf to hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/archive/data/default/TestAcidGuarantees/a29fa882a9e3f50c391e3aa08c5c5e72/A/454801a3e8174a35b153f39efae8f0cf 2023-04-15 02:55:56,411 DEBUG [HFileArchiver-11] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/.tmp/data/default/TestAcidGuarantees/a29fa882a9e3f50c391e3aa08c5c5e72/A/6ec73ecb341c44509a59650277eb89c1 to hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/archive/data/default/TestAcidGuarantees/a29fa882a9e3f50c391e3aa08c5c5e72/A/6ec73ecb341c44509a59650277eb89c1 2023-04-15 02:55:56,412 DEBUG [HFileArchiver-11] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/.tmp/data/default/TestAcidGuarantees/a29fa882a9e3f50c391e3aa08c5c5e72/A/8e57c5759618496b882a08e61652edbc to hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/archive/data/default/TestAcidGuarantees/a29fa882a9e3f50c391e3aa08c5c5e72/A/8e57c5759618496b882a08e61652edbc 2023-04-15 02:55:56,415 DEBUG [HFileArchiver-11] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/.tmp/data/default/TestAcidGuarantees/a29fa882a9e3f50c391e3aa08c5c5e72/B/2c8a05257dd04ce7a656c414cd1c6fae to hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/archive/data/default/TestAcidGuarantees/a29fa882a9e3f50c391e3aa08c5c5e72/B/2c8a05257dd04ce7a656c414cd1c6fae 2023-04-15 02:55:56,416 DEBUG [HFileArchiver-11] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/.tmp/data/default/TestAcidGuarantees/a29fa882a9e3f50c391e3aa08c5c5e72/B/6ca987c0fd8f4e8a9389deba06e5447d to hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/archive/data/default/TestAcidGuarantees/a29fa882a9e3f50c391e3aa08c5c5e72/B/6ca987c0fd8f4e8a9389deba06e5447d 2023-04-15 02:55:56,418 DEBUG [HFileArchiver-11] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/.tmp/data/default/TestAcidGuarantees/a29fa882a9e3f50c391e3aa08c5c5e72/B/ef39994cb7ec48cd9fb92f26b4698b26 to hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/archive/data/default/TestAcidGuarantees/a29fa882a9e3f50c391e3aa08c5c5e72/B/ef39994cb7ec48cd9fb92f26b4698b26 2023-04-15 02:55:56,421 DEBUG [HFileArchiver-11] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/.tmp/data/default/TestAcidGuarantees/a29fa882a9e3f50c391e3aa08c5c5e72/C/1ccd90366ab74a97944655816a237ecd to hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/archive/data/default/TestAcidGuarantees/a29fa882a9e3f50c391e3aa08c5c5e72/C/1ccd90366ab74a97944655816a237ecd 2023-04-15 02:55:56,423 DEBUG [HFileArchiver-11] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/.tmp/data/default/TestAcidGuarantees/a29fa882a9e3f50c391e3aa08c5c5e72/C/88d606c95a3e4d27b7727f29fa316b53 to hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/archive/data/default/TestAcidGuarantees/a29fa882a9e3f50c391e3aa08c5c5e72/C/88d606c95a3e4d27b7727f29fa316b53 2023-04-15 02:55:56,425 DEBUG [HFileArchiver-11] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/.tmp/data/default/TestAcidGuarantees/a29fa882a9e3f50c391e3aa08c5c5e72/C/c07ea2e71580443c9ae8193743672c05 to hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/archive/data/default/TestAcidGuarantees/a29fa882a9e3f50c391e3aa08c5c5e72/C/c07ea2e71580443c9ae8193743672c05 2023-04-15 02:55:56,429 DEBUG [HFileArchiver-11] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/.tmp/data/default/TestAcidGuarantees/a29fa882a9e3f50c391e3aa08c5c5e72/recovered.edits/508.seqid to hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/archive/data/default/TestAcidGuarantees/a29fa882a9e3f50c391e3aa08c5c5e72/recovered.edits/508.seqid 2023-04-15 02:55:56,430 DEBUG [HFileArchiver-11] backup.HFileArchiver(596): Deleted hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/.tmp/data/default/TestAcidGuarantees/a29fa882a9e3f50c391e3aa08c5c5e72 2023-04-15 02:55:56,430 DEBUG [PEWorker-2] procedure.DeleteTableProcedure(328): Archived TestAcidGuarantees regions 2023-04-15 02:55:56,434 DEBUG [PEWorker-2] procedure.DeleteTableProcedure(118): Deleting regions from META for pid=53, state=RUNNABLE:DELETE_TABLE_REMOVE_FROM_META, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-15 02:55:56,436 WARN [PEWorker-2] procedure.DeleteTableProcedure(384): Deleting some vestigial 1 rows of TestAcidGuarantees from hbase:meta 2023-04-15 02:55:56,438 DEBUG [PEWorker-2] procedure.DeleteTableProcedure(421): Removing 'TestAcidGuarantees' descriptor. 2023-04-15 02:55:56,440 DEBUG [PEWorker-2] procedure.DeleteTableProcedure(124): Deleting assignment state for pid=53, state=RUNNABLE:DELETE_TABLE_UNASSIGN_REGIONS, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-15 02:55:56,440 DEBUG [PEWorker-2] procedure.DeleteTableProcedure(411): Removing 'TestAcidGuarantees' from region states. 2023-04-15 02:55:56,440 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"TestAcidGuarantees,,1681527331158.a29fa882a9e3f50c391e3aa08c5c5e72.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1681527356440"}]},"ts":"9223372036854775807"} 2023-04-15 02:55:56,443 INFO [PEWorker-2] hbase.MetaTableAccessor(1788): Deleted 1 regions from META 2023-04-15 02:55:56,443 DEBUG [PEWorker-2] hbase.MetaTableAccessor(1789): Deleted regions: [{ENCODED => a29fa882a9e3f50c391e3aa08c5c5e72, NAME => 'TestAcidGuarantees,,1681527331158.a29fa882a9e3f50c391e3aa08c5c5e72.', STARTKEY => '', ENDKEY => ''}] 2023-04-15 02:55:56,443 DEBUG [PEWorker-2] procedure.DeleteTableProcedure(415): Marking 'TestAcidGuarantees' as deleted. 2023-04-15 02:55:56,443 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":0,"tag":[],"timestamp":"1681527356443"}]},"ts":"9223372036854775807"} 2023-04-15 02:55:56,445 INFO [PEWorker-2] hbase.MetaTableAccessor(1658): Deleted table TestAcidGuarantees state from META 2023-04-15 02:55:56,448 DEBUG [PEWorker-2] procedure.DeleteTableProcedure(130): Finished pid=53, state=RUNNABLE:DELETE_TABLE_POST_OPERATION, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-15 02:55:56,449 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=53, state=SUCCESS; DeleteTableProcedure table=TestAcidGuarantees in 58 msec 2023-04-15 02:55:56,496 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(1227): Checking to see if procedure is done pid=53 2023-04-15 02:55:56,496 INFO [Listener at localhost.localdomain/33693] client.HBaseAdmin$TableFuture(3541): Operation: DELETE, Table Name: default:TestAcidGuarantees, procId: 53 completed 2023-04-15 02:55:56,514 INFO [Listener at localhost.localdomain/33693] hbase.ResourceChecker(175): after: TestAcidGuaranteesWithEagerPolicy#testScanAtomicity Thread=230 (was 228) - Thread LEAK? -, OpenFileDescriptor=537 (was 528) - OpenFileDescriptor LEAK? -, MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=362 (was 344) - SystemLoadAverage LEAK? -, ProcessCount=168 (was 168), AvailableMemoryMB=5681 (was 5606) - AvailableMemoryMB LEAK? - 2023-04-15 02:55:56,524 INFO [Listener at localhost.localdomain/33693] hbase.ResourceChecker(147): before: TestAcidGuaranteesWithEagerPolicy#testMobGetAtomicity Thread=230, OpenFileDescriptor=537, MaxFileDescriptor=60000, SystemLoadAverage=362, ProcessCount=168, AvailableMemoryMB=5681 2023-04-15 02:55:56,526 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] util.TableDescriptorChecker(340): MEMSTORE_FLUSHSIZE for table descriptor or "hbase.hregion.memstore.flush.size" (131072) is too small, which might cause very frequent flushing. 2023-04-15 02:55:56,526 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.HMaster$4(2112): Client=jenkins//148.251.75.209 create 'TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-04-15 02:55:56,527 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] procedure2.ProcedureExecutor(1029): Stored pid=54, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION; CreateTableProcedure table=TestAcidGuarantees 2023-04-15 02:55:56,528 INFO [PEWorker-5] procedure.CreateTableProcedure(80): pid=54, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_PRE_OPERATION 2023-04-15 02:55:56,528 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(697): Client=jenkins//148.251.75.209 procedure request for creating table: namespace: "default" qualifier: "TestAcidGuarantees" procId is: 54 2023-04-15 02:55:56,529 INFO [PEWorker-5] procedure.CreateTableProcedure(80): pid=54, state=RUNNABLE:CREATE_TABLE_WRITE_FS_LAYOUT, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_WRITE_FS_LAYOUT 2023-04-15 02:55:56,529 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(1227): Checking to see if procedure is done pid=54 2023-04-15 02:55:56,530 DEBUG [HFileArchiver-12] backup.HFileArchiver(131): ARCHIVING hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/.tmp/data/default/TestAcidGuarantees/20e761db5975fb9082f870bf056bd078 2023-04-15 02:55:56,530 DEBUG [HFileArchiver-12] backup.HFileArchiver(153): Directory hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/.tmp/data/default/TestAcidGuarantees/20e761db5975fb9082f870bf056bd078 empty. 2023-04-15 02:55:56,531 DEBUG [HFileArchiver-12] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/.tmp/data/default/TestAcidGuarantees/20e761db5975fb9082f870bf056bd078 2023-04-15 02:55:56,531 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(328): Archived TestAcidGuarantees regions 2023-04-15 02:55:56,631 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(1227): Checking to see if procedure is done pid=54 2023-04-15 02:55:56,834 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(1227): Checking to see if procedure is done pid=54 2023-04-15 02:55:56,939 DEBUG [PEWorker-5] util.FSTableDescriptors(570): Wrote into hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/.tmp/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000001 2023-04-15 02:55:56,941 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(7675): creating {ENCODED => 20e761db5975fb9082f870bf056bd078, NAME => 'TestAcidGuarantees,,1681527356526.20e761db5975fb9082f870bf056bd078.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/.tmp 2023-04-15 02:55:56,950 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1681527356526.20e761db5975fb9082f870bf056bd078.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-15 02:55:56,950 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1604): Closing 20e761db5975fb9082f870bf056bd078, disabling compactions & flushes 2023-04-15 02:55:56,950 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1681527356526.20e761db5975fb9082f870bf056bd078. 2023-04-15 02:55:56,950 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1681527356526.20e761db5975fb9082f870bf056bd078. 2023-04-15 02:55:56,950 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1681527356526.20e761db5975fb9082f870bf056bd078. after waiting 0 ms 2023-04-15 02:55:56,950 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1681527356526.20e761db5975fb9082f870bf056bd078. 2023-04-15 02:55:56,951 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1681527356526.20e761db5975fb9082f870bf056bd078. 2023-04-15 02:55:56,951 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1558): Region close journal for 20e761db5975fb9082f870bf056bd078: 2023-04-15 02:55:56,954 INFO [PEWorker-5] procedure.CreateTableProcedure(80): pid=54, state=RUNNABLE:CREATE_TABLE_ADD_TO_META, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_ADD_TO_META 2023-04-15 02:55:56,955 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1681527356526.20e761db5975fb9082f870bf056bd078.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1681527356955"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1681527356955"}]},"ts":"1681527356955"} 2023-04-15 02:55:56,956 INFO [PEWorker-5] hbase.MetaTableAccessor(1496): Added 1 regions to meta. 2023-04-15 02:55:56,956 INFO [PEWorker-5] procedure.CreateTableProcedure(80): pid=54, state=RUNNABLE:CREATE_TABLE_ASSIGN_REGIONS, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_ASSIGN_REGIONS 2023-04-15 02:55:56,956 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1681527356956"}]},"ts":"1681527356956"} 2023-04-15 02:55:56,957 INFO [PEWorker-5] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=ENABLING in hbase:meta 2023-04-15 02:55:56,959 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=55, ppid=54, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=TestAcidGuarantees, region=20e761db5975fb9082f870bf056bd078, ASSIGN}] 2023-04-15 02:55:56,960 INFO [PEWorker-4] procedure.MasterProcedureScheduler(727): Took xlock for pid=55, ppid=54, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=TestAcidGuarantees, region=20e761db5975fb9082f870bf056bd078, ASSIGN 2023-04-15 02:55:56,961 INFO [PEWorker-4] assignment.TransitRegionStateProcedure(193): Starting pid=55, ppid=54, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=TestAcidGuarantees, region=20e761db5975fb9082f870bf056bd078, ASSIGN; state=OFFLINE, location=jenkins-hbase20.apache.org,34681,1681527209133; forceNewPlan=false, retain=false 2023-04-15 02:55:57,113 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=55 updating hbase:meta row=20e761db5975fb9082f870bf056bd078, regionState=OPENING, regionLocation=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:57,114 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1681527356526.20e761db5975fb9082f870bf056bd078.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1681527357113"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1681527357113"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1681527357113"}]},"ts":"1681527357113"} 2023-04-15 02:55:57,117 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=56, ppid=55, state=RUNNABLE; OpenRegionProcedure 20e761db5975fb9082f870bf056bd078, server=jenkins-hbase20.apache.org,34681,1681527209133}] 2023-04-15 02:55:57,136 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(1227): Checking to see if procedure is done pid=54 2023-04-15 02:55:57,276 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] handler.AssignRegionHandler(128): Open TestAcidGuarantees,,1681527356526.20e761db5975fb9082f870bf056bd078. 2023-04-15 02:55:57,277 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => 20e761db5975fb9082f870bf056bd078, NAME => 'TestAcidGuarantees,,1681527356526.20e761db5975fb9082f870bf056bd078.', STARTKEY => '', ENDKEY => ''} 2023-04-15 02:55:57,277 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table TestAcidGuarantees 20e761db5975fb9082f870bf056bd078 2023-04-15 02:55:57,277 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1681527356526.20e761db5975fb9082f870bf056bd078.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-15 02:55:57,277 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7894): checking encryption for 20e761db5975fb9082f870bf056bd078 2023-04-15 02:55:57,278 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7897): checking classloading for 20e761db5975fb9082f870bf056bd078 2023-04-15 02:55:57,280 INFO [StoreOpener-20e761db5975fb9082f870bf056bd078-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family A of region 20e761db5975fb9082f870bf056bd078 2023-04-15 02:55:57,283 DEBUG [StoreOpener-20e761db5975fb9082f870bf056bd078-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/20e761db5975fb9082f870bf056bd078/A 2023-04-15 02:55:57,283 DEBUG [StoreOpener-20e761db5975fb9082f870bf056bd078-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/20e761db5975fb9082f870bf056bd078/A 2023-04-15 02:55:57,284 INFO [StoreOpener-20e761db5975fb9082f870bf056bd078-1] regionserver.CompactingMemStore(121): Store=A, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-15 02:55:57,284 INFO [StoreOpener-20e761db5975fb9082f870bf056bd078-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 20e761db5975fb9082f870bf056bd078 columnFamilyName A 2023-04-15 02:55:57,285 INFO [StoreOpener-20e761db5975fb9082f870bf056bd078-1] regionserver.HStore(310): Store=20e761db5975fb9082f870bf056bd078/A, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-15 02:55:57,285 INFO [StoreOpener-20e761db5975fb9082f870bf056bd078-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family B of region 20e761db5975fb9082f870bf056bd078 2023-04-15 02:55:57,287 DEBUG [StoreOpener-20e761db5975fb9082f870bf056bd078-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/20e761db5975fb9082f870bf056bd078/B 2023-04-15 02:55:57,287 DEBUG [StoreOpener-20e761db5975fb9082f870bf056bd078-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/20e761db5975fb9082f870bf056bd078/B 2023-04-15 02:55:57,287 INFO [StoreOpener-20e761db5975fb9082f870bf056bd078-1] regionserver.CompactingMemStore(121): Store=B, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-15 02:55:57,287 INFO [StoreOpener-20e761db5975fb9082f870bf056bd078-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 20e761db5975fb9082f870bf056bd078 columnFamilyName B 2023-04-15 02:55:57,288 INFO [StoreOpener-20e761db5975fb9082f870bf056bd078-1] regionserver.HStore(310): Store=20e761db5975fb9082f870bf056bd078/B, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-15 02:55:57,288 INFO [StoreOpener-20e761db5975fb9082f870bf056bd078-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family C of region 20e761db5975fb9082f870bf056bd078 2023-04-15 02:55:57,290 DEBUG [StoreOpener-20e761db5975fb9082f870bf056bd078-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/20e761db5975fb9082f870bf056bd078/C 2023-04-15 02:55:57,290 DEBUG [StoreOpener-20e761db5975fb9082f870bf056bd078-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/20e761db5975fb9082f870bf056bd078/C 2023-04-15 02:55:57,290 INFO [StoreOpener-20e761db5975fb9082f870bf056bd078-1] regionserver.CompactingMemStore(121): Store=C, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-15 02:55:57,290 INFO [StoreOpener-20e761db5975fb9082f870bf056bd078-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 20e761db5975fb9082f870bf056bd078 columnFamilyName C 2023-04-15 02:55:57,291 INFO [StoreOpener-20e761db5975fb9082f870bf056bd078-1] regionserver.HStore(310): Store=20e761db5975fb9082f870bf056bd078/C, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-15 02:55:57,291 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1146): Setting FlushNonSloppyStoresFirstPolicy for the region=TestAcidGuarantees,,1681527356526.20e761db5975fb9082f870bf056bd078. 2023-04-15 02:55:57,292 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/20e761db5975fb9082f870bf056bd078 2023-04-15 02:55:57,293 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/20e761db5975fb9082f870bf056bd078 2023-04-15 02:55:57,296 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table TestAcidGuarantees descriptor;using region.getMemStoreFlushHeapSize/# of families (16.0 M)) instead. 2023-04-15 02:55:57,297 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1055): writing seq id for 20e761db5975fb9082f870bf056bd078 2023-04-15 02:55:57,300 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/20e761db5975fb9082f870bf056bd078/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-04-15 02:55:57,300 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1072): Opened 20e761db5975fb9082f870bf056bd078; next sequenceid=2; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=64841536, jitterRate=-0.03378582000732422}, FlushLargeStoresPolicy{flushSizeLowerBound=16777216} 2023-04-15 02:55:57,300 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(965): Region open journal for 20e761db5975fb9082f870bf056bd078: 2023-04-15 02:55:57,301 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for TestAcidGuarantees,,1681527356526.20e761db5975fb9082f870bf056bd078., pid=56, masterSystemTime=1681527357269 2023-04-15 02:55:57,303 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for TestAcidGuarantees,,1681527356526.20e761db5975fb9082f870bf056bd078. 2023-04-15 02:55:57,303 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] handler.AssignRegionHandler(156): Opened TestAcidGuarantees,,1681527356526.20e761db5975fb9082f870bf056bd078. 2023-04-15 02:55:57,304 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=55 updating hbase:meta row=20e761db5975fb9082f870bf056bd078, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:57,304 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"TestAcidGuarantees,,1681527356526.20e761db5975fb9082f870bf056bd078.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1681527357304"},{"qualifier":"server","vlen":32,"tag":[],"timestamp":"1681527357304"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1681527357304"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1681527357304"}]},"ts":"1681527357304"} 2023-04-15 02:55:57,308 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=56, resume processing ppid=55 2023-04-15 02:55:57,308 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=56, ppid=55, state=SUCCESS; OpenRegionProcedure 20e761db5975fb9082f870bf056bd078, server=jenkins-hbase20.apache.org,34681,1681527209133 in 190 msec 2023-04-15 02:55:57,311 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=55, resume processing ppid=54 2023-04-15 02:55:57,311 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=55, ppid=54, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=20e761db5975fb9082f870bf056bd078, ASSIGN in 349 msec 2023-04-15 02:55:57,312 INFO [PEWorker-4] procedure.CreateTableProcedure(80): pid=54, state=RUNNABLE:CREATE_TABLE_UPDATE_DESC_CACHE, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_UPDATE_DESC_CACHE 2023-04-15 02:55:57,312 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1681527357312"}]},"ts":"1681527357312"} 2023-04-15 02:55:57,313 INFO [PEWorker-4] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=ENABLED in hbase:meta 2023-04-15 02:55:57,316 INFO [PEWorker-4] procedure.CreateTableProcedure(80): pid=54, state=RUNNABLE:CREATE_TABLE_POST_OPERATION, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_POST_OPERATION 2023-04-15 02:55:57,318 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=54, state=SUCCESS; CreateTableProcedure table=TestAcidGuarantees in 790 msec 2023-04-15 02:55:57,638 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(1227): Checking to see if procedure is done pid=54 2023-04-15 02:55:57,638 INFO [Listener at localhost.localdomain/33693] client.HBaseAdmin$TableFuture(3541): Operation: CREATE, Table Name: default:TestAcidGuarantees, procId: 54 completed 2023-04-15 02:55:57,640 DEBUG [Listener at localhost.localdomain/33693] zookeeper.ReadOnlyZKClient(139): Connect 0x65a9ccaf to 127.0.0.1:51762 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-15 02:55:57,649 DEBUG [Listener at localhost.localdomain/33693] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@9b5ade7, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-15 02:55:57,651 DEBUG [Listener at localhost.localdomain/33693] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-15 02:55:57,652 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:48584, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-15 02:55:57,653 DEBUG [Listener at localhost.localdomain/33693] ipc.RpcConnection(124): Using SIMPLE authentication for service=MasterService, sasl=false 2023-04-15 02:55:57,656 INFO [RS-EventLoopGroup-1-3] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:50248, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=MasterService 2023-04-15 02:55:57,658 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] util.TableDescriptorChecker(340): MEMSTORE_FLUSHSIZE for table descriptor or "hbase.hregion.memstore.flush.size" (131072) is too small, which might cause very frequent flushing. 2023-04-15 02:55:57,659 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.HMaster$12(2454): Client=jenkins//148.251.75.209 modify table TestAcidGuarantees from 'TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} to 'TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', MOB_THRESHOLD => '4', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-04-15 02:55:57,661 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] procedure2.ProcedureExecutor(1029): Stored pid=57, state=RUNNABLE:MODIFY_TABLE_PREPARE; ModifyTableProcedure table=TestAcidGuarantees 2023-04-15 02:55:58,072 DEBUG [PEWorker-1] util.FSTableDescriptors(570): Wrote into hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000002 2023-04-15 02:55:58,074 DEBUG [PEWorker-1] util.FSTableDescriptors(519): Deleted hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000001 2023-04-15 02:55:58,075 INFO [PEWorker-1] util.FSTableDescriptors(293): Updated tableinfo=hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000002 2023-04-15 02:55:58,079 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=58, ppid=57, state=RUNNABLE:REOPEN_TABLE_REGIONS_GET_REGIONS; ReopenTableRegionsProcedure table=TestAcidGuarantees}] 2023-04-15 02:55:58,082 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=59, ppid=58, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=20e761db5975fb9082f870bf056bd078, REOPEN/MOVE}] 2023-04-15 02:55:58,083 INFO [PEWorker-2] procedure.MasterProcedureScheduler(727): Took xlock for pid=59, ppid=58, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=20e761db5975fb9082f870bf056bd078, REOPEN/MOVE 2023-04-15 02:55:58,084 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=59 updating hbase:meta row=20e761db5975fb9082f870bf056bd078, regionState=CLOSING, regionLocation=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:58,084 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1681527356526.20e761db5975fb9082f870bf056bd078.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1681527358084"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1681527358084"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1681527358084"}]},"ts":"1681527358084"} 2023-04-15 02:55:58,085 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=60, ppid=59, state=RUNNABLE; CloseRegionProcedure 20e761db5975fb9082f870bf056bd078, server=jenkins-hbase20.apache.org,34681,1681527209133}] 2023-04-15 02:55:58,237 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] handler.UnassignRegionHandler(109): Close 20e761db5975fb9082f870bf056bd078 2023-04-15 02:55:58,239 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1604): Closing 20e761db5975fb9082f870bf056bd078, disabling compactions & flushes 2023-04-15 02:55:58,239 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1681527356526.20e761db5975fb9082f870bf056bd078. 2023-04-15 02:55:58,239 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1681527356526.20e761db5975fb9082f870bf056bd078. 2023-04-15 02:55:58,239 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1681527356526.20e761db5975fb9082f870bf056bd078. after waiting 0 ms 2023-04-15 02:55:58,239 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1681527356526.20e761db5975fb9082f870bf056bd078. 2023-04-15 02:55:58,247 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/20e761db5975fb9082f870bf056bd078/recovered.edits/4.seqid, newMaxSeqId=4, maxSeqId=1 2023-04-15 02:55:58,248 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1681527356526.20e761db5975fb9082f870bf056bd078. 2023-04-15 02:55:58,248 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1558): Region close journal for 20e761db5975fb9082f870bf056bd078: 2023-04-15 02:55:58,248 WARN [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionServer(3509): Not adding moved region record: 20e761db5975fb9082f870bf056bd078 to self. 2023-04-15 02:55:58,250 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] handler.UnassignRegionHandler(147): Closed 20e761db5975fb9082f870bf056bd078 2023-04-15 02:55:58,250 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=59 updating hbase:meta row=20e761db5975fb9082f870bf056bd078, regionState=CLOSED 2023-04-15 02:55:58,250 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1681527356526.20e761db5975fb9082f870bf056bd078.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1681527358250"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1681527358250"}]},"ts":"1681527358250"} 2023-04-15 02:55:58,253 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=60, resume processing ppid=59 2023-04-15 02:55:58,254 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=60, ppid=59, state=SUCCESS; CloseRegionProcedure 20e761db5975fb9082f870bf056bd078, server=jenkins-hbase20.apache.org,34681,1681527209133 in 166 msec 2023-04-15 02:55:58,254 INFO [PEWorker-1] assignment.TransitRegionStateProcedure(193): Starting pid=59, ppid=58, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=TestAcidGuarantees, region=20e761db5975fb9082f870bf056bd078, REOPEN/MOVE; state=CLOSED, location=jenkins-hbase20.apache.org,34681,1681527209133; forceNewPlan=false, retain=true 2023-04-15 02:55:58,405 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=59 updating hbase:meta row=20e761db5975fb9082f870bf056bd078, regionState=OPENING, regionLocation=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:58,405 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1681527356526.20e761db5975fb9082f870bf056bd078.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1681527358405"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1681527358405"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1681527358405"}]},"ts":"1681527358405"} 2023-04-15 02:55:58,409 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=61, ppid=59, state=RUNNABLE; OpenRegionProcedure 20e761db5975fb9082f870bf056bd078, server=jenkins-hbase20.apache.org,34681,1681527209133}] 2023-04-15 02:55:58,569 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] handler.AssignRegionHandler(128): Open TestAcidGuarantees,,1681527356526.20e761db5975fb9082f870bf056bd078. 2023-04-15 02:55:58,569 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => 20e761db5975fb9082f870bf056bd078, NAME => 'TestAcidGuarantees,,1681527356526.20e761db5975fb9082f870bf056bd078.', STARTKEY => '', ENDKEY => ''} 2023-04-15 02:55:58,570 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table TestAcidGuarantees 20e761db5975fb9082f870bf056bd078 2023-04-15 02:55:58,570 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1681527356526.20e761db5975fb9082f870bf056bd078.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-15 02:55:58,570 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7894): checking encryption for 20e761db5975fb9082f870bf056bd078 2023-04-15 02:55:58,570 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7897): checking classloading for 20e761db5975fb9082f870bf056bd078 2023-04-15 02:55:58,572 INFO [StoreOpener-20e761db5975fb9082f870bf056bd078-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family A of region 20e761db5975fb9082f870bf056bd078 2023-04-15 02:55:58,574 DEBUG [StoreOpener-20e761db5975fb9082f870bf056bd078-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/20e761db5975fb9082f870bf056bd078/A 2023-04-15 02:55:58,574 DEBUG [StoreOpener-20e761db5975fb9082f870bf056bd078-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/20e761db5975fb9082f870bf056bd078/A 2023-04-15 02:55:58,574 INFO [StoreOpener-20e761db5975fb9082f870bf056bd078-1] regionserver.CompactingMemStore(121): Store=A, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-15 02:55:58,575 INFO [StoreOpener-20e761db5975fb9082f870bf056bd078-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 20e761db5975fb9082f870bf056bd078 columnFamilyName A 2023-04-15 02:55:58,577 INFO [StoreOpener-20e761db5975fb9082f870bf056bd078-1] regionserver.HStore(310): Store=20e761db5975fb9082f870bf056bd078/A, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-15 02:55:58,578 INFO [StoreOpener-20e761db5975fb9082f870bf056bd078-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family B of region 20e761db5975fb9082f870bf056bd078 2023-04-15 02:55:58,579 DEBUG [StoreOpener-20e761db5975fb9082f870bf056bd078-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/20e761db5975fb9082f870bf056bd078/B 2023-04-15 02:55:58,579 DEBUG [StoreOpener-20e761db5975fb9082f870bf056bd078-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/20e761db5975fb9082f870bf056bd078/B 2023-04-15 02:55:58,580 INFO [StoreOpener-20e761db5975fb9082f870bf056bd078-1] regionserver.CompactingMemStore(121): Store=B, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-15 02:55:58,580 INFO [StoreOpener-20e761db5975fb9082f870bf056bd078-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 20e761db5975fb9082f870bf056bd078 columnFamilyName B 2023-04-15 02:55:58,581 INFO [StoreOpener-20e761db5975fb9082f870bf056bd078-1] regionserver.HStore(310): Store=20e761db5975fb9082f870bf056bd078/B, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-15 02:55:58,581 INFO [StoreOpener-20e761db5975fb9082f870bf056bd078-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family C of region 20e761db5975fb9082f870bf056bd078 2023-04-15 02:55:58,583 DEBUG [StoreOpener-20e761db5975fb9082f870bf056bd078-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/20e761db5975fb9082f870bf056bd078/C 2023-04-15 02:55:58,583 DEBUG [StoreOpener-20e761db5975fb9082f870bf056bd078-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/20e761db5975fb9082f870bf056bd078/C 2023-04-15 02:55:58,583 INFO [StoreOpener-20e761db5975fb9082f870bf056bd078-1] regionserver.CompactingMemStore(121): Store=C, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-15 02:55:58,584 INFO [StoreOpener-20e761db5975fb9082f870bf056bd078-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 20e761db5975fb9082f870bf056bd078 columnFamilyName C 2023-04-15 02:55:58,584 INFO [StoreOpener-20e761db5975fb9082f870bf056bd078-1] regionserver.HStore(310): Store=20e761db5975fb9082f870bf056bd078/C, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-15 02:55:58,585 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1146): Setting FlushNonSloppyStoresFirstPolicy for the region=TestAcidGuarantees,,1681527356526.20e761db5975fb9082f870bf056bd078. 2023-04-15 02:55:58,585 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/20e761db5975fb9082f870bf056bd078 2023-04-15 02:55:58,586 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/20e761db5975fb9082f870bf056bd078 2023-04-15 02:55:58,589 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table TestAcidGuarantees descriptor;using region.getMemStoreFlushHeapSize/# of families (16.0 M)) instead. 2023-04-15 02:55:58,590 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1055): writing seq id for 20e761db5975fb9082f870bf056bd078 2023-04-15 02:55:58,592 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1072): Opened 20e761db5975fb9082f870bf056bd078; next sequenceid=5; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=67825504, jitterRate=0.010678768157958984}, FlushLargeStoresPolicy{flushSizeLowerBound=16777216} 2023-04-15 02:55:58,592 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(965): Region open journal for 20e761db5975fb9082f870bf056bd078: 2023-04-15 02:55:58,592 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for TestAcidGuarantees,,1681527356526.20e761db5975fb9082f870bf056bd078., pid=61, masterSystemTime=1681527358562 2023-04-15 02:55:58,594 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for TestAcidGuarantees,,1681527356526.20e761db5975fb9082f870bf056bd078. 2023-04-15 02:55:58,594 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] handler.AssignRegionHandler(156): Opened TestAcidGuarantees,,1681527356526.20e761db5975fb9082f870bf056bd078. 2023-04-15 02:55:58,594 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=59 updating hbase:meta row=20e761db5975fb9082f870bf056bd078, regionState=OPEN, openSeqNum=5, regionLocation=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:58,594 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"TestAcidGuarantees,,1681527356526.20e761db5975fb9082f870bf056bd078.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1681527358594"},{"qualifier":"server","vlen":32,"tag":[],"timestamp":"1681527358594"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1681527358594"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1681527358594"}]},"ts":"1681527358594"} 2023-04-15 02:55:58,596 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=61, resume processing ppid=59 2023-04-15 02:55:58,596 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=61, ppid=59, state=SUCCESS; OpenRegionProcedure 20e761db5975fb9082f870bf056bd078, server=jenkins-hbase20.apache.org,34681,1681527209133 in 187 msec 2023-04-15 02:55:58,598 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=59, resume processing ppid=58 2023-04-15 02:55:58,598 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=59, ppid=58, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=20e761db5975fb9082f870bf056bd078, REOPEN/MOVE in 514 msec 2023-04-15 02:55:58,600 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=58, resume processing ppid=57 2023-04-15 02:55:58,600 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=58, ppid=57, state=SUCCESS; ReopenTableRegionsProcedure table=TestAcidGuarantees in 519 msec 2023-04-15 02:55:58,602 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=57, state=SUCCESS; ModifyTableProcedure table=TestAcidGuarantees in 942 msec 2023-04-15 02:55:58,603 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(1227): Checking to see if procedure is done pid=57 2023-04-15 02:55:58,603 DEBUG [Listener at localhost.localdomain/33693] zookeeper.ReadOnlyZKClient(139): Connect 0x03b84625 to 127.0.0.1:51762 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-15 02:55:58,607 DEBUG [Listener at localhost.localdomain/33693] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@285f2e49, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-15 02:55:58,608 DEBUG [Listener at localhost.localdomain/33693] zookeeper.ReadOnlyZKClient(139): Connect 0x4ae8f5b1 to 127.0.0.1:51762 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-15 02:55:58,611 DEBUG [Listener at localhost.localdomain/33693] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@b200ba7, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-15 02:55:58,612 DEBUG [Listener at localhost.localdomain/33693] zookeeper.ReadOnlyZKClient(139): Connect 0x5ddb3e63 to 127.0.0.1:51762 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-15 02:55:58,617 DEBUG [Listener at localhost.localdomain/33693] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@50131d14, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-15 02:55:58,617 DEBUG [Listener at localhost.localdomain/33693] zookeeper.ReadOnlyZKClient(139): Connect 0x630ea023 to 127.0.0.1:51762 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-15 02:55:58,621 DEBUG [Listener at localhost.localdomain/33693] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@6722b207, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-15 02:55:58,621 DEBUG [Listener at localhost.localdomain/33693] zookeeper.ReadOnlyZKClient(139): Connect 0x5ec98d63 to 127.0.0.1:51762 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-15 02:55:58,638 DEBUG [Listener at localhost.localdomain/33693] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@48502121, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-15 02:55:58,638 DEBUG [Listener at localhost.localdomain/33693] zookeeper.ReadOnlyZKClient(139): Connect 0x08d3d8bc to 127.0.0.1:51762 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-15 02:55:58,642 DEBUG [Listener at localhost.localdomain/33693] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@4143b9ae, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-15 02:55:58,643 DEBUG [Listener at localhost.localdomain/33693] zookeeper.ReadOnlyZKClient(139): Connect 0x7e6ad437 to 127.0.0.1:51762 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-15 02:55:58,646 DEBUG [Listener at localhost.localdomain/33693] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@686af57e, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-15 02:55:58,647 DEBUG [Listener at localhost.localdomain/33693] zookeeper.ReadOnlyZKClient(139): Connect 0x2f6f7187 to 127.0.0.1:51762 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-15 02:55:58,665 DEBUG [Listener at localhost.localdomain/33693] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@78cd331, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-15 02:55:58,665 DEBUG [Listener at localhost.localdomain/33693] zookeeper.ReadOnlyZKClient(139): Connect 0x49c34fbe to 127.0.0.1:51762 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-15 02:55:58,673 DEBUG [Listener at localhost.localdomain/33693] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@1d74fc48, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-15 02:55:58,674 DEBUG [Listener at localhost.localdomain/33693] zookeeper.ReadOnlyZKClient(139): Connect 0x5467be78 to 127.0.0.1:51762 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-15 02:55:58,681 DEBUG [Listener at localhost.localdomain/33693] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@70ab5f5, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-15 02:55:58,691 DEBUG [hconnection-0x27c0140d-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-15 02:55:58,691 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-15 02:55:58,692 DEBUG [hconnection-0x70f3e066-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-15 02:55:58,692 DEBUG [hconnection-0x46edc6f1-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-15 02:55:58,693 DEBUG [hconnection-0x69c46ced-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-15 02:55:58,694 DEBUG [hconnection-0x1b3df336-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-15 02:55:58,695 DEBUG [hconnection-0x2cef8da7-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-15 02:55:58,695 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:48612, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-15 02:55:58,695 DEBUG [hconnection-0x3f4cfc0-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-15 02:55:58,696 DEBUG [hconnection-0x31ea9869-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-15 02:55:58,698 DEBUG [hconnection-0x51ec1052-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-15 02:55:58,698 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:48642, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-15 02:55:58,698 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:48598, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-15 02:55:58,699 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:48626, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-15 02:55:58,701 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:48652, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-15 02:55:58,701 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:48664, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-15 02:55:58,702 DEBUG [hconnection-0x6cc7df84-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-15 02:55:58,707 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:48586, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-15 02:55:58,709 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:48624, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-15 02:55:58,709 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:48648, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-15 02:55:58,709 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-15 02:55:58,709 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-15 02:55:58,709 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-15 02:55:58,710 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-15 02:55:58,711 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-15 02:55:58,711 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-15 02:55:58,711 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:48666, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-15 02:55:58,712 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:58,712 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:55:58,713 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:58,713 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-15 02:55:58,713 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-15 02:55:58,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-15 02:55:58,713 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:58,713 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-15 02:55:58,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:55:58,713 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:58,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-15 02:55:58,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:55:58,715 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-15 02:55:58,718 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-15 02:55:58,719 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34681] regionserver.HRegion(9158): Flush requested on 20e761db5975fb9082f870bf056bd078 2023-04-15 02:55:58,719 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 20e761db5975fb9082f870bf056bd078 3/3 column families, dataSize=60.38 KB heapSize=180.05 KB 2023-04-15 02:55:58,719 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 20e761db5975fb9082f870bf056bd078, store=A 2023-04-15 02:55:58,720 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-15 02:55:58,720 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 20e761db5975fb9082f870bf056bd078, store=B 2023-04-15 02:55:58,720 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-15 02:55:58,720 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 20e761db5975fb9082f870bf056bd078, store=C 2023-04-15 02:55:58,720 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-15 02:55:58,719 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-15 02:55:58,720 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-15 02:55:58,720 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-15 02:55:58,720 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-15 02:55:58,720 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-15 02:55:58,720 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-15 02:55:58,720 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681527356526.20e761db5975fb9082f870bf056bd078. 2023-04-15 02:55:58,720 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681527356526.20e761db5975fb9082f870bf056bd078. started... 2023-04-15 02:55:58,721 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681527356526.20e761db5975fb9082f870bf056bd078. as already flushing 2023-04-15 02:55:58,721 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681527356526.20e761db5975fb9082f870bf056bd078. 2023-04-15 02:55:58,721 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-15 02:55:58,721 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-15 02:55:58,721 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:58,721 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-15 02:55:58,721 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34681,1681527209133' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-15 02:55:58,722 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:58,722 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:58,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:58,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:55:58,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:55:58,722 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:58,722 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-15 02:55:58,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:55:58,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:55:58,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:55:58,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:58,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:55:58,723 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34681,1681527209133' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-15 02:55:58,723 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6748d06d[Count = 0] remaining members to acquire global barrier 2023-04-15 02:55:58,723 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-15 02:55:58,723 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:58,724 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:58,724 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:58,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:58,724 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-15 02:55:58,724 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-15 02:55:58,724 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:58,724 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-15 02:55:58,724 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34681,1681527209133' in zk 2023-04-15 02:55:58,725 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:58,725 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-15 02:55:58,725 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-15 02:55:58,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:58,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:55:58,725 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-15 02:55:58,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:55:58,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:55:58,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:55:58,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:55:58,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:58,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:55:58,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:55:58,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:58,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34681,1681527209133': 2023-04-15 02:55:58,728 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34681,1681527209133' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-15 02:55:58,728 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-15 02:55:58,728 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-15 02:55:58,728 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-15 02:55:58,728 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-15 02:55:58,728 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-15 02:55:58,728 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:58,728 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:58,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:58,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:55:58,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:55:58,729 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-15 02:55:58,729 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:58,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:58,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:55:58,729 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-15 02:55:58,729 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:58,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-15 02:55:58,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:58,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:55:58,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:55:58,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:55:58,730 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:58,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:58,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:55:58,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:55:58,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:58,732 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:58,732 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:55:58,732 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:55:58,732 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-15 02:55:58,732 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-15 02:55:58,732 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-15 02:55:58,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-15 02:55:58,734 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-15 02:55:58,734 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-15 02:55:58,734 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-15 02:55:58,734 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-15 02:55:58,734 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-15 02:55:58,734 DEBUG [Thread-3439] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-15 02:55:58,734 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:58,735 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:58,735 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:58,735 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:58,735 DEBUG [Thread-3439] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-15 02:55:58,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-15 02:55:58,739 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230415b159c1baef0e41639ac587235c017ee1 to hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230415b159c1baef0e41639ac587235c017ee1 2023-04-15 02:55:58,740 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9564, totalHeapSize=21976, min timestamp=1681527358730, max timestamp=1681527358739Num uniques 859; 2023-04-15 02:55:58,740 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-4] regionserver.CompactionPipeline(163): Suffix data size=39334, new segment data size=9564, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-15 02:55:58,741 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9564, totalHeapSize=21976, min timestamp=1681527358730, max timestamp=1681527358739Num uniques 859; 2023-04-15 02:55:58,741 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-8] regionserver.CompactionPipeline(163): Suffix data size=39334, new segment data size=9564, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-15 02:55:58,742 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9564, totalHeapSize=21976, min timestamp=1681527358730, max timestamp=1681527358739Num uniques 859; 2023-04-15 02:55:58,742 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-4] regionserver.CompactionPipeline(163): Suffix data size=39334, new segment data size=9564, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-15 02:55:58,747 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=16, memsize=20.1 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/20e761db5975fb9082f870bf056bd078/.tmp/A/a714ed51fc7c46378b9e8630cea6f95f 2023-04-15 02:55:58,752 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=20e761db5975fb9082f870bf056bd078, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:55:58,752 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=20e761db5975fb9082f870bf056bd078, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:55:58,752 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34681] ipc.CallRunner(144): callId: 10 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:48586 deadline: 1681527418749, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=20e761db5975fb9082f870bf056bd078, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:58,752 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34681] ipc.CallRunner(144): callId: 9 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:48666 deadline: 1681527418749, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=20e761db5975fb9082f870bf056bd078, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:58,753 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=20e761db5975fb9082f870bf056bd078, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:55:58,753 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34681] ipc.CallRunner(144): callId: 11 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:48664 deadline: 1681527418751, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=20e761db5975fb9082f870bf056bd078, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:58,754 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=20e761db5975fb9082f870bf056bd078, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:55:58,754 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34681] ipc.CallRunner(144): callId: 10 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:48624 deadline: 1681527418752, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=20e761db5975fb9082f870bf056bd078, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:58,754 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=20e761db5975fb9082f870bf056bd078, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:55:58,754 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34681] ipc.CallRunner(144): callId: 10 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:48648 deadline: 1681527418752, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=20e761db5975fb9082f870bf056bd078, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:58,835 DEBUG [Thread-3439] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-15 02:55:58,835 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-15 02:55:58,837 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-15 02:55:58,838 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-15 02:55:58,838 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-15 02:55:58,838 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-15 02:55:58,838 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-15 02:55:58,838 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-15 02:55:58,838 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-15 02:55:58,838 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:58,838 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:55:58,839 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:58,839 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-15 02:55:58,839 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-15 02:55:58,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-15 02:55:58,839 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:58,839 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-15 02:55:58,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:55:58,839 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:58,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-15 02:55:58,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:55:58,840 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-15 02:55:58,840 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-15 02:55:58,840 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-15 02:55:58,840 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-15 02:55:58,840 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-15 02:55:58,840 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-15 02:55:58,840 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-15 02:55:58,840 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-15 02:55:58,840 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681527356526.20e761db5975fb9082f870bf056bd078. 2023-04-15 02:55:58,840 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681527356526.20e761db5975fb9082f870bf056bd078. started... 2023-04-15 02:55:58,840 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681527356526.20e761db5975fb9082f870bf056bd078. as already flushing 2023-04-15 02:55:58,841 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681527356526.20e761db5975fb9082f870bf056bd078. 2023-04-15 02:55:58,841 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-15 02:55:58,841 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-15 02:55:58,841 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:58,841 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-15 02:55:58,841 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34681,1681527209133' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-15 02:55:58,841 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:58,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:58,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:55:58,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:55:58,842 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:58,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:55:58,842 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:58,842 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-15 02:55:58,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:55:58,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:55:58,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:58,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:55:58,843 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34681,1681527209133' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-15 02:55:58,843 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4a7df0cf[Count = 0] remaining members to acquire global barrier 2023-04-15 02:55:58,843 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-15 02:55:58,843 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:58,844 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:58,844 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:58,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:58,844 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-15 02:55:58,844 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-15 02:55:58,844 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34681,1681527209133' in zk 2023-04-15 02:55:58,845 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:58,845 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-15 02:55:58,845 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:58,845 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-15 02:55:58,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:58,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:55:58,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:55:58,845 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-15 02:55:58,845 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-15 02:55:58,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:55:58,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:55:58,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:55:58,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:58,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:55:58,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:55:58,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:58,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34681,1681527209133': 2023-04-15 02:55:58,848 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34681,1681527209133' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-15 02:55:58,848 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-15 02:55:58,848 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-15 02:55:58,848 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-15 02:55:58,848 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-15 02:55:58,848 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-15 02:55:58,848 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:58,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:58,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:55:58,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:55:58,849 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:58,849 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-15 02:55:58,849 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:58,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:58,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:55:58,849 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-15 02:55:58,849 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:58,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-15 02:55:58,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:55:58,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:58,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:55:58,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:55:58,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:58,851 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:58,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:55:58,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:55:58,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:58,852 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-15 02:55:58,852 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-15 02:55:58,852 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-15 02:55:58,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-15 02:55:58,852 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:58,852 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-15 02:55:58,852 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-15 02:55:58,852 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-15 02:55:58,852 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:55:58,852 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:55:58,852 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:58,852 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-15 02:55:58,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-15 02:55:58,852 DEBUG [Thread-3439] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-15 02:55:58,852 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:58,853 DEBUG [Thread-3439] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-15 02:55:58,853 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:58,853 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:58,854 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=20e761db5975fb9082f870bf056bd078, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:55:58,854 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=20e761db5975fb9082f870bf056bd078, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:55:58,854 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34681] ipc.CallRunner(144): callId: 11 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:48666 deadline: 1681527418853, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=20e761db5975fb9082f870bf056bd078, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:58,854 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34681] ipc.CallRunner(144): callId: 12 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:48586 deadline: 1681527418853, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=20e761db5975fb9082f870bf056bd078, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:58,855 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=20e761db5975fb9082f870bf056bd078, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:55:58,855 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34681] ipc.CallRunner(144): callId: 13 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:48664 deadline: 1681527418854, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=20e761db5975fb9082f870bf056bd078, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:58,855 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=20e761db5975fb9082f870bf056bd078, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:55:58,855 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34681] ipc.CallRunner(144): callId: 12 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:48624 deadline: 1681527418854, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=20e761db5975fb9082f870bf056bd078, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:58,855 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=20e761db5975fb9082f870bf056bd078, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:55:58,855 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34681] ipc.CallRunner(144): callId: 12 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:48648 deadline: 1681527418854, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=20e761db5975fb9082f870bf056bd078, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:58,953 DEBUG [Thread-3439] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-15 02:55:58,953 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-15 02:55:58,954 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-15 02:55:58,955 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-15 02:55:58,955 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-15 02:55:58,955 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-15 02:55:58,955 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-15 02:55:58,955 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-15 02:55:58,955 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-15 02:55:58,956 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:58,956 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:55:58,956 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:58,956 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-15 02:55:58,956 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-15 02:55:58,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-15 02:55:58,956 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:58,956 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-15 02:55:58,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:55:58,957 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:58,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-15 02:55:58,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:55:58,957 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-15 02:55:58,957 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-15 02:55:58,957 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-15 02:55:58,957 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-15 02:55:58,957 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-15 02:55:58,957 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-15 02:55:58,957 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-15 02:55:58,957 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681527356526.20e761db5975fb9082f870bf056bd078. 2023-04-15 02:55:58,957 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681527356526.20e761db5975fb9082f870bf056bd078. started... 2023-04-15 02:55:58,958 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681527356526.20e761db5975fb9082f870bf056bd078. as already flushing 2023-04-15 02:55:58,958 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681527356526.20e761db5975fb9082f870bf056bd078. 2023-04-15 02:55:58,958 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-15 02:55:58,958 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-15 02:55:58,958 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:58,958 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-15 02:55:58,958 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34681,1681527209133' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-15 02:55:58,959 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:58,959 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:58,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:58,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:55:58,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:55:58,959 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:58,959 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-15 02:55:58,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:55:58,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:55:58,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:55:58,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:58,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:55:58,961 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34681,1681527209133' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-15 02:55:58,961 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-15 02:55:58,961 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@50f23975[Count = 0] remaining members to acquire global barrier 2023-04-15 02:55:58,961 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:58,961 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:58,961 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:58,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:58,961 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:58,961 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-15 02:55:58,961 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-15 02:55:58,961 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-15 02:55:58,962 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34681,1681527209133' in zk 2023-04-15 02:55:58,962 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:58,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:58,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:55:58,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:55:58,962 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-15 02:55:58,962 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-15 02:55:58,962 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-15 02:55:58,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:55:58,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:55:58,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:55:58,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:58,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:55:58,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:55:58,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:58,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34681,1681527209133': 2023-04-15 02:55:58,964 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34681,1681527209133' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-15 02:55:58,964 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-15 02:55:58,964 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-15 02:55:58,964 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-15 02:55:58,964 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-15 02:55:58,964 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-15 02:55:58,965 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:58,965 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:58,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:58,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:55:58,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:55:58,965 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:58,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:58,965 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-15 02:55:58,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:55:58,965 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-15 02:55:58,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-15 02:55:58,966 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:58,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:58,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:55:58,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:55:58,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:55:58,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:58,966 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:58,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:55:58,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:55:58,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:58,968 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:58,968 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:55:58,968 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-15 02:55:58,968 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-15 02:55:58,968 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-15 02:55:58,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-15 02:55:58,968 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-15 02:55:58,968 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-15 02:55:58,968 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-15 02:55:58,968 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-15 02:55:58,968 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:55:58,969 DEBUG [Thread-3439] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-15 02:55:58,969 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-15 02:55:58,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-15 02:55:58,969 DEBUG [Thread-3439] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-15 02:55:58,969 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:58,969 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:58,969 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:58,969 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:59,057 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=20e761db5975fb9082f870bf056bd078, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:55:59,057 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=20e761db5975fb9082f870bf056bd078, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:55:59,057 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34681] ipc.CallRunner(144): callId: 15 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:48664 deadline: 1681527419056, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=20e761db5975fb9082f870bf056bd078, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:59,057 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34681] ipc.CallRunner(144): callId: 14 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:48586 deadline: 1681527419056, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=20e761db5975fb9082f870bf056bd078, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:59,057 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=20e761db5975fb9082f870bf056bd078, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:55:59,057 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=20e761db5975fb9082f870bf056bd078, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:55:59,057 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34681] ipc.CallRunner(144): callId: 14 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:48624 deadline: 1681527419056, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=20e761db5975fb9082f870bf056bd078, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:59,057 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34681] ipc.CallRunner(144): callId: 13 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:48666 deadline: 1681527419056, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=20e761db5975fb9082f870bf056bd078, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:59,058 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=20e761db5975fb9082f870bf056bd078, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:55:59,058 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34681] ipc.CallRunner(144): callId: 14 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:48648 deadline: 1681527419057, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=20e761db5975fb9082f870bf056bd078, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:59,069 DEBUG [Thread-3439] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-15 02:55:59,069 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-15 02:55:59,070 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-15 02:55:59,071 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-15 02:55:59,071 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-15 02:55:59,071 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-15 02:55:59,072 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-15 02:55:59,072 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-15 02:55:59,072 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-15 02:55:59,072 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:59,072 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:55:59,073 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:59,073 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-15 02:55:59,073 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-15 02:55:59,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-15 02:55:59,073 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:59,073 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-15 02:55:59,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:55:59,073 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:59,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-15 02:55:59,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:55:59,073 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-15 02:55:59,074 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-15 02:55:59,074 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-15 02:55:59,074 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-15 02:55:59,074 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-15 02:55:59,074 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-15 02:55:59,074 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-15 02:55:59,074 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-15 02:55:59,074 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681527356526.20e761db5975fb9082f870bf056bd078. 2023-04-15 02:55:59,074 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681527356526.20e761db5975fb9082f870bf056bd078. started... 2023-04-15 02:55:59,074 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681527356526.20e761db5975fb9082f870bf056bd078. as already flushing 2023-04-15 02:55:59,074 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681527356526.20e761db5975fb9082f870bf056bd078. 2023-04-15 02:55:59,074 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-15 02:55:59,074 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-15 02:55:59,074 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:59,074 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-15 02:55:59,074 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34681,1681527209133' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-15 02:55:59,075 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:59,075 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:59,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:59,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:55:59,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:55:59,075 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:59,076 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-15 02:55:59,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:55:59,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:55:59,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:55:59,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:59,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:55:59,077 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34681,1681527209133' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-15 02:55:59,077 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-15 02:55:59,077 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7974b682[Count = 0] remaining members to acquire global barrier 2023-04-15 02:55:59,077 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:59,078 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:59,078 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:59,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:59,078 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-15 02:55:59,078 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-15 02:55:59,078 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34681,1681527209133' in zk 2023-04-15 02:55:59,078 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:59,078 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-15 02:55:59,080 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:59,081 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-15 02:55:59,081 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-15 02:55:59,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:59,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:55:59,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:55:59,081 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-15 02:55:59,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:55:59,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:55:59,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:55:59,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:59,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:55:59,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:55:59,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:59,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34681,1681527209133': 2023-04-15 02:55:59,082 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34681,1681527209133' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-15 02:55:59,082 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-15 02:55:59,082 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-15 02:55:59,082 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-15 02:55:59,082 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-15 02:55:59,082 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-15 02:55:59,084 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:59,084 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:59,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:59,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:55:59,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:55:59,084 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:59,084 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-15 02:55:59,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:59,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:55:59,084 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:59,084 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-15 02:55:59,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-15 02:55:59,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:55:59,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:59,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:55:59,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:55:59,085 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:59,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:59,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:55:59,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:55:59,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:59,086 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:59,086 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:55:59,086 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:55:59,086 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:59,086 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-15 02:55:59,086 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-15 02:55:59,086 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-15 02:55:59,086 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-15 02:55:59,086 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:59,086 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-15 02:55:59,086 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-15 02:55:59,087 DEBUG [Thread-3439] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-15 02:55:59,086 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:55:59,087 DEBUG [Thread-3439] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-15 02:55:59,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-15 02:55:59,087 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:59,087 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-15 02:55:59,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-15 02:55:59,160 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=16 (bloomFilter=true), to=hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/20e761db5975fb9082f870bf056bd078/.tmp/B/ce7e96538ce54fc2bed1e90c526b8f0e 2023-04-15 02:55:59,168 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=16 (bloomFilter=true), to=hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/20e761db5975fb9082f870bf056bd078/.tmp/C/bc0770e966664656b1a39dc2b7bc02ad 2023-04-15 02:55:59,171 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/20e761db5975fb9082f870bf056bd078/.tmp/A/a714ed51fc7c46378b9e8630cea6f95f as hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/20e761db5975fb9082f870bf056bd078/A/a714ed51fc7c46378b9e8630cea6f95f 2023-04-15 02:55:59,174 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/20e761db5975fb9082f870bf056bd078/A/a714ed51fc7c46378b9e8630cea6f95f, entries=150, sequenceid=16, filesize=25.1 K 2023-04-15 02:55:59,174 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/20e761db5975fb9082f870bf056bd078/.tmp/B/ce7e96538ce54fc2bed1e90c526b8f0e as hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/20e761db5975fb9082f870bf056bd078/B/ce7e96538ce54fc2bed1e90c526b8f0e 2023-04-15 02:55:59,177 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/20e761db5975fb9082f870bf056bd078/B/ce7e96538ce54fc2bed1e90c526b8f0e, entries=150, sequenceid=16, filesize=11.6 K 2023-04-15 02:55:59,178 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/20e761db5975fb9082f870bf056bd078/.tmp/C/bc0770e966664656b1a39dc2b7bc02ad as hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/20e761db5975fb9082f870bf056bd078/C/bc0770e966664656b1a39dc2b7bc02ad 2023-04-15 02:55:59,180 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/20e761db5975fb9082f870bf056bd078/C/bc0770e966664656b1a39dc2b7bc02ad, entries=150, sequenceid=16, filesize=11.6 K 2023-04-15 02:55:59,181 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~60.38 KB/61830, heapSize ~180 KB/184320, currentSize=120.76 KB/123660 for 20e761db5975fb9082f870bf056bd078 in 462ms, sequenceid=16, compaction requested=false 2023-04-15 02:55:59,181 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 20e761db5975fb9082f870bf056bd078: 2023-04-15 02:55:59,187 DEBUG [Thread-3439] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-15 02:55:59,187 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-15 02:55:59,188 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-15 02:55:59,189 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-15 02:55:59,189 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-15 02:55:59,189 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-15 02:55:59,189 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-15 02:55:59,189 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-15 02:55:59,189 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-15 02:55:59,190 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:59,190 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:55:59,190 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-15 02:55:59,190 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:59,190 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-15 02:55:59,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-15 02:55:59,191 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:59,191 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-15 02:55:59,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:55:59,191 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:55:59,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-15 02:55:59,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:55:59,191 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-15 02:55:59,191 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-15 02:55:59,191 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-15 02:55:59,191 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-15 02:55:59,191 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-15 02:55:59,191 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-15 02:55:59,192 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-15 02:55:59,192 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-15 02:55:59,192 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681527356526.20e761db5975fb9082f870bf056bd078. 2023-04-15 02:55:59,192 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681527356526.20e761db5975fb9082f870bf056bd078. started... 2023-04-15 02:55:59,192 INFO [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 20e761db5975fb9082f870bf056bd078 3/3 column families, dataSize=120.76 KB heapSize=340.45 KB 2023-04-15 02:55:59,192 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 20e761db5975fb9082f870bf056bd078, store=A 2023-04-15 02:55:59,192 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-15 02:55:59,192 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 20e761db5975fb9082f870bf056bd078, store=B 2023-04-15 02:55:59,192 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-15 02:55:59,192 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 20e761db5975fb9082f870bf056bd078, store=C 2023-04-15 02:55:59,192 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-15 02:55:59,204 INFO [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230415f96606d3392c4f5cb5058787f9e075d6 to hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230415f96606d3392c4f5cb5058787f9e075d6 2023-04-15 02:55:59,360 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34681] regionserver.HRegion(9158): Flush requested on 20e761db5975fb9082f870bf056bd078 2023-04-15 02:55:59,360 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681527356526.20e761db5975fb9082f870bf056bd078. as already flushing 2023-04-15 02:55:59,367 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=20e761db5975fb9082f870bf056bd078, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:55:59,367 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34681] ipc.CallRunner(144): callId: 18 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:48648 deadline: 1681527419364, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=20e761db5975fb9082f870bf056bd078, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:59,367 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=20e761db5975fb9082f870bf056bd078, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:55:59,367 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=20e761db5975fb9082f870bf056bd078, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:55:59,367 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34681] ipc.CallRunner(144): callId: 18 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:48586 deadline: 1681527419365, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=20e761db5975fb9082f870bf056bd078, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:59,368 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34681] ipc.CallRunner(144): callId: 18 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:48624 deadline: 1681527419365, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=20e761db5975fb9082f870bf056bd078, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:59,368 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=20e761db5975fb9082f870bf056bd078, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:55:59,368 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34681] ipc.CallRunner(144): callId: 17 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:48666 deadline: 1681527419365, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=20e761db5975fb9082f870bf056bd078, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:59,368 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=20e761db5975fb9082f870bf056bd078, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:55:59,368 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34681] ipc.CallRunner(144): callId: 19 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:48664 deadline: 1681527419366, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=20e761db5975fb9082f870bf056bd078, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:59,469 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=20e761db5975fb9082f870bf056bd078, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:55:59,469 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34681] ipc.CallRunner(144): callId: 20 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:48648 deadline: 1681527419468, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=20e761db5975fb9082f870bf056bd078, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:59,469 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=20e761db5975fb9082f870bf056bd078, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:55:59,469 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34681] ipc.CallRunner(144): callId: 20 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:48586 deadline: 1681527419468, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=20e761db5975fb9082f870bf056bd078, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:59,470 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=20e761db5975fb9082f870bf056bd078, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:55:59,470 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34681] ipc.CallRunner(144): callId: 19 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:48666 deadline: 1681527419468, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=20e761db5975fb9082f870bf056bd078, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:59,470 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=20e761db5975fb9082f870bf056bd078, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:55:59,470 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34681] ipc.CallRunner(144): callId: 21 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:48664 deadline: 1681527419468, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=20e761db5975fb9082f870bf056bd078, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:59,472 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=20e761db5975fb9082f870bf056bd078, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:55:59,472 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34681] ipc.CallRunner(144): callId: 20 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:48624 deadline: 1681527419470, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=20e761db5975fb9082f870bf056bd078, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:59,608 INFO [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=50, memsize=40.3 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/20e761db5975fb9082f870bf056bd078/.tmp/A/78ddfefdb20c4498b0be0f13b8b9bf97 2023-04-15 02:55:59,671 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=20e761db5975fb9082f870bf056bd078, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:55:59,671 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34681] ipc.CallRunner(144): callId: 22 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:48586 deadline: 1681527419670, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=20e761db5975fb9082f870bf056bd078, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:59,671 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=20e761db5975fb9082f870bf056bd078, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:55:59,672 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34681] ipc.CallRunner(144): callId: 22 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:48648 deadline: 1681527419671, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=20e761db5975fb9082f870bf056bd078, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:59,672 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=20e761db5975fb9082f870bf056bd078, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:55:59,672 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34681] ipc.CallRunner(144): callId: 21 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:48666 deadline: 1681527419671, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=20e761db5975fb9082f870bf056bd078, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:59,673 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=20e761db5975fb9082f870bf056bd078, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:55:59,673 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34681] ipc.CallRunner(144): callId: 23 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:48664 deadline: 1681527419671, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=20e761db5975fb9082f870bf056bd078, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:59,674 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=20e761db5975fb9082f870bf056bd078, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:55:59,675 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34681] ipc.CallRunner(144): callId: 22 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:48624 deadline: 1681527419673, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=20e761db5975fb9082f870bf056bd078, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:59,973 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=20e761db5975fb9082f870bf056bd078, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:55:59,973 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34681] ipc.CallRunner(144): callId: 24 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:48586 deadline: 1681527419972, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=20e761db5975fb9082f870bf056bd078, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:59,973 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=20e761db5975fb9082f870bf056bd078, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:55:59,973 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34681] ipc.CallRunner(144): callId: 24 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:48648 deadline: 1681527419972, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=20e761db5975fb9082f870bf056bd078, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:59,976 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=20e761db5975fb9082f870bf056bd078, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:55:59,976 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34681] ipc.CallRunner(144): callId: 23 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:48666 deadline: 1681527419974, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=20e761db5975fb9082f870bf056bd078, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:59,976 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=20e761db5975fb9082f870bf056bd078, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:55:59,976 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34681] ipc.CallRunner(144): callId: 25 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:48664 deadline: 1681527419976, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=20e761db5975fb9082f870bf056bd078, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:55:59,978 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=20e761db5975fb9082f870bf056bd078, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:55:59,978 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34681] ipc.CallRunner(144): callId: 24 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:48624 deadline: 1681527419977, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=20e761db5975fb9082f870bf056bd078, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:00,016 INFO [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=40.25 KB at sequenceid=50 (bloomFilter=true), to=hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/20e761db5975fb9082f870bf056bd078/.tmp/B/a78ad28d40df4cb39afa5886b9b0cf03 2023-04-15 02:56:00,428 INFO [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=40.25 KB at sequenceid=50 (bloomFilter=true), to=hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/20e761db5975fb9082f870bf056bd078/.tmp/C/3d55dce025b5415982c8518bb01bd7d4 2023-04-15 02:56:00,431 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/20e761db5975fb9082f870bf056bd078/.tmp/A/78ddfefdb20c4498b0be0f13b8b9bf97 as hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/20e761db5975fb9082f870bf056bd078/A/78ddfefdb20c4498b0be0f13b8b9bf97 2023-04-15 02:56:00,434 INFO [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/20e761db5975fb9082f870bf056bd078/A/78ddfefdb20c4498b0be0f13b8b9bf97, entries=150, sequenceid=50, filesize=25.1 K 2023-04-15 02:56:00,434 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/20e761db5975fb9082f870bf056bd078/.tmp/B/a78ad28d40df4cb39afa5886b9b0cf03 as hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/20e761db5975fb9082f870bf056bd078/B/a78ad28d40df4cb39afa5886b9b0cf03 2023-04-15 02:56:00,437 INFO [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/20e761db5975fb9082f870bf056bd078/B/a78ad28d40df4cb39afa5886b9b0cf03, entries=150, sequenceid=50, filesize=11.6 K 2023-04-15 02:56:00,437 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/20e761db5975fb9082f870bf056bd078/.tmp/C/3d55dce025b5415982c8518bb01bd7d4 as hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/20e761db5975fb9082f870bf056bd078/C/3d55dce025b5415982c8518bb01bd7d4 2023-04-15 02:56:00,440 INFO [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/20e761db5975fb9082f870bf056bd078/C/3d55dce025b5415982c8518bb01bd7d4, entries=150, sequenceid=50, filesize=11.6 K 2023-04-15 02:56:00,440 INFO [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~120.76 KB/123660, heapSize ~340.41 KB/348576, currentSize=67.09 KB/68700 for 20e761db5975fb9082f870bf056bd078 in 1248ms, sequenceid=50, compaction requested=false 2023-04-15 02:56:00,440 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 20e761db5975fb9082f870bf056bd078: 2023-04-15 02:56:00,440 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681527356526.20e761db5975fb9082f870bf056bd078. 2023-04-15 02:56:00,440 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-15 02:56:00,440 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-15 02:56:00,440 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:00,441 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-15 02:56:00,441 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34681,1681527209133' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-15 02:56:00,441 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:56:00,442 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:00,442 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:56:00,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:00,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:56:00,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:56:00,442 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-15 02:56:00,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:56:00,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:56:00,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:56:00,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:00,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:56:00,443 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34681,1681527209133' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-15 02:56:00,443 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-15 02:56:00,443 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@589eda79[Count = 0] remaining members to acquire global barrier 2023-04-15 02:56:00,443 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:56:00,443 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:56:00,443 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:56:00,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:56:00,443 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-15 02:56:00,443 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-15 02:56:00,444 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34681,1681527209133' in zk 2023-04-15 02:56:00,444 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:00,444 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-15 02:56:00,444 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:00,444 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-15 02:56:00,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:00,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:56:00,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:56:00,444 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-15 02:56:00,444 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-15 02:56:00,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:56:00,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:56:00,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:56:00,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:00,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:56:00,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:56:00,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:00,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34681,1681527209133': 2023-04-15 02:56:00,446 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34681,1681527209133' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-15 02:56:00,446 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-15 02:56:00,446 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-15 02:56:00,446 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-15 02:56:00,446 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-15 02:56:00,446 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-15 02:56:00,447 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:56:00,447 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:56:00,447 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:56:00,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:56:00,447 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-15 02:56:00,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:56:00,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:56:00,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:56:00,447 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-15 02:56:00,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-15 02:56:00,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:56:00,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:56:00,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:56:00,448 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:00,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:56:00,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:56:00,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:00,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:56:00,449 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:00,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:56:00,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:00,450 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:00,450 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:56:00,450 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:56:00,450 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:00,450 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:56:00,450 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-15 02:56:00,450 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-15 02:56:00,450 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-15 02:56:00,450 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:56:00,450 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-15 02:56:00,450 DEBUG [Thread-3439] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-15 02:56:00,450 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:56:00,450 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-15 02:56:00,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-15 02:56:00,450 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-15 02:56:00,450 DEBUG [Thread-3439] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-15 02:56:00,451 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-15 02:56:00,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-15 02:56:00,479 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34681] regionserver.HRegion(9158): Flush requested on 20e761db5975fb9082f870bf056bd078 2023-04-15 02:56:00,479 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 20e761db5975fb9082f870bf056bd078 3/3 column families, dataSize=73.80 KB heapSize=219.89 KB 2023-04-15 02:56:00,479 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 20e761db5975fb9082f870bf056bd078, store=A 2023-04-15 02:56:00,479 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-15 02:56:00,479 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 20e761db5975fb9082f870bf056bd078, store=B 2023-04-15 02:56:00,479 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-15 02:56:00,479 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 20e761db5975fb9082f870bf056bd078, store=C 2023-04-15 02:56:00,479 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-15 02:56:00,488 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023041523555c1aa3954069828223303aece98d to hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023041523555c1aa3954069828223303aece98d 2023-04-15 02:56:00,491 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=20e761db5975fb9082f870bf056bd078, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:56:00,491 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34681] ipc.CallRunner(144): callId: 27 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:48666 deadline: 1681527420488, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=20e761db5975fb9082f870bf056bd078, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:00,493 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=20e761db5975fb9082f870bf056bd078, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:56:00,493 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34681] ipc.CallRunner(144): callId: 30 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:48586 deadline: 1681527420491, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=20e761db5975fb9082f870bf056bd078, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:00,493 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=20e761db5975fb9082f870bf056bd078, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:56:00,494 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34681] ipc.CallRunner(144): callId: 30 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:48664 deadline: 1681527420491, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=20e761db5975fb9082f870bf056bd078, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:00,494 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=20e761db5975fb9082f870bf056bd078, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:56:00,494 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34681] ipc.CallRunner(144): callId: 31 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:48648 deadline: 1681527420491, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=20e761db5975fb9082f870bf056bd078, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:00,494 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=20e761db5975fb9082f870bf056bd078, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:56:00,494 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34681] ipc.CallRunner(144): callId: 29 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:48624 deadline: 1681527420492, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=20e761db5975fb9082f870bf056bd078, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:00,551 DEBUG [Thread-3439] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-15 02:56:00,551 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-15 02:56:00,552 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-15 02:56:00,553 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-15 02:56:00,553 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-15 02:56:00,553 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-15 02:56:00,553 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-15 02:56:00,553 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-15 02:56:00,553 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-15 02:56:00,553 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:56:00,554 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:56:00,554 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-15 02:56:00,554 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:00,554 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-15 02:56:00,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-15 02:56:00,554 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:00,554 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-15 02:56:00,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:56:00,555 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:56:00,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-15 02:56:00,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:56:00,555 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-15 02:56:00,555 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-15 02:56:00,555 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-15 02:56:00,555 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-15 02:56:00,555 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-15 02:56:00,555 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-15 02:56:00,555 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-15 02:56:00,555 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681527356526.20e761db5975fb9082f870bf056bd078. 2023-04-15 02:56:00,555 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681527356526.20e761db5975fb9082f870bf056bd078. started... 2023-04-15 02:56:00,555 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681527356526.20e761db5975fb9082f870bf056bd078. as already flushing 2023-04-15 02:56:00,555 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681527356526.20e761db5975fb9082f870bf056bd078. 2023-04-15 02:56:00,556 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-15 02:56:00,556 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-15 02:56:00,556 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:00,556 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-15 02:56:00,556 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34681,1681527209133' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-15 02:56:00,556 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:00,556 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:56:00,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:00,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:56:00,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:56:00,557 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:56:00,557 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-15 02:56:00,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:56:00,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:56:00,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:56:00,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:00,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:56:00,558 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34681,1681527209133' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-15 02:56:00,558 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7dc0b22d[Count = 0] remaining members to acquire global barrier 2023-04-15 02:56:00,558 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-15 02:56:00,558 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:56:00,558 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:56:00,558 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:56:00,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:56:00,558 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-15 02:56:00,558 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-15 02:56:00,558 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34681,1681527209133' in zk 2023-04-15 02:56:00,558 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:00,558 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-15 02:56:00,559 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:00,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:00,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:56:00,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:56:00,559 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-15 02:56:00,559 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-15 02:56:00,559 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-15 02:56:00,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:56:00,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:56:00,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:56:00,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:00,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:56:00,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:56:00,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:00,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34681,1681527209133': 2023-04-15 02:56:00,561 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34681,1681527209133' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-15 02:56:00,561 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-15 02:56:00,561 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-15 02:56:00,561 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-15 02:56:00,561 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-15 02:56:00,561 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-15 02:56:00,561 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:56:00,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:56:00,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:56:00,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:56:00,561 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:56:00,562 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-15 02:56:00,562 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:56:00,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:56:00,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:56:00,562 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-15 02:56:00,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-15 02:56:00,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:56:00,562 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:00,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:56:00,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:56:00,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:56:00,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:00,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:56:00,563 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:00,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:56:00,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:00,565 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-15 02:56:00,565 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-15 02:56:00,565 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:00,565 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-15 02:56:00,565 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-15 02:56:00,565 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:56:00,565 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-15 02:56:00,565 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:56:00,565 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-15 02:56:00,565 DEBUG [Thread-3439] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-15 02:56:00,565 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:00,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-15 02:56:00,565 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:56:00,565 DEBUG [Thread-3439] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-15 02:56:00,565 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:56:00,565 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:56:00,565 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-15 02:56:00,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-15 02:56:00,595 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=20e761db5975fb9082f870bf056bd078, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:56:00,595 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34681] ipc.CallRunner(144): callId: 29 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:48666 deadline: 1681527420592, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=20e761db5975fb9082f870bf056bd078, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:00,596 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=20e761db5975fb9082f870bf056bd078, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:56:00,596 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34681] ipc.CallRunner(144): callId: 32 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:48586 deadline: 1681527420594, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=20e761db5975fb9082f870bf056bd078, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:00,597 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=20e761db5975fb9082f870bf056bd078, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:56:00,597 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=20e761db5975fb9082f870bf056bd078, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:56:00,597 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34681] ipc.CallRunner(144): callId: 32 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:48664 deadline: 1681527420594, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=20e761db5975fb9082f870bf056bd078, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:00,597 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34681] ipc.CallRunner(144): callId: 33 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:48648 deadline: 1681527420594, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=20e761db5975fb9082f870bf056bd078, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:00,597 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=20e761db5975fb9082f870bf056bd078, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:56:00,597 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34681] ipc.CallRunner(144): callId: 31 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:48624 deadline: 1681527420594, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=20e761db5975fb9082f870bf056bd078, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:00,665 DEBUG [Thread-3439] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-15 02:56:00,666 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-15 02:56:00,667 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-15 02:56:00,667 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-15 02:56:00,667 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-15 02:56:00,667 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-15 02:56:00,667 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-15 02:56:00,667 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-15 02:56:00,667 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-15 02:56:00,668 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:56:00,668 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:56:00,669 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:00,669 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-15 02:56:00,669 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-15 02:56:00,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-15 02:56:00,669 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:00,669 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-15 02:56:00,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:56:00,670 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:56:00,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-15 02:56:00,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:56:00,670 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-15 02:56:00,670 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-15 02:56:00,670 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-15 02:56:00,670 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-15 02:56:00,670 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-15 02:56:00,670 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-15 02:56:00,671 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-15 02:56:00,671 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-15 02:56:00,671 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681527356526.20e761db5975fb9082f870bf056bd078. 2023-04-15 02:56:00,671 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681527356526.20e761db5975fb9082f870bf056bd078. started... 2023-04-15 02:56:00,671 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681527356526.20e761db5975fb9082f870bf056bd078. as already flushing 2023-04-15 02:56:00,671 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681527356526.20e761db5975fb9082f870bf056bd078. 2023-04-15 02:56:00,671 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-15 02:56:00,671 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-15 02:56:00,671 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:00,671 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-15 02:56:00,671 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34681,1681527209133' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-15 02:56:00,672 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:00,672 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:56:00,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:00,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:56:00,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:56:00,672 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:56:00,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:56:00,672 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-15 02:56:00,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:56:00,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:56:00,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:00,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:56:00,673 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34681,1681527209133' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-15 02:56:00,673 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3cb81571[Count = 0] remaining members to acquire global barrier 2023-04-15 02:56:00,673 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-15 02:56:00,673 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:56:00,674 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:56:00,674 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:56:00,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:56:00,674 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-15 02:56:00,674 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-15 02:56:00,674 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34681,1681527209133' in zk 2023-04-15 02:56:00,674 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:00,674 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-15 02:56:00,674 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:00,674 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-15 02:56:00,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:00,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:56:00,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:56:00,674 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-15 02:56:00,675 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-15 02:56:00,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:56:00,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:56:00,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:56:00,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:00,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:56:00,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:56:00,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:00,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34681,1681527209133': 2023-04-15 02:56:00,676 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34681,1681527209133' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-15 02:56:00,676 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-15 02:56:00,676 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-15 02:56:00,676 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-15 02:56:00,676 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-15 02:56:00,676 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-15 02:56:00,677 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:56:00,677 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-15 02:56:00,677 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:56:00,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:56:00,677 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:56:00,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:56:00,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:56:00,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:56:00,677 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-15 02:56:00,677 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:00,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-15 02:56:00,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:56:00,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:56:00,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:56:00,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:56:00,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:56:00,678 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:00,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:00,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:56:00,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:56:00,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:00,679 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:00,679 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-15 02:56:00,679 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:56:00,679 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:56:00,679 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:00,679 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-15 02:56:00,679 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-15 02:56:00,679 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-15 02:56:00,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-15 02:56:00,679 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-15 02:56:00,679 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:56:00,679 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-15 02:56:00,680 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-15 02:56:00,680 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:56:00,680 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:56:00,680 DEBUG [Thread-3439] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-15 02:56:00,680 DEBUG [Thread-3439] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-15 02:56:00,680 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-15 02:56:00,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-15 02:56:00,780 DEBUG [Thread-3439] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-15 02:56:00,780 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-15 02:56:00,782 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-15 02:56:00,783 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-15 02:56:00,783 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-15 02:56:00,783 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-15 02:56:00,783 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-15 02:56:00,783 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-15 02:56:00,783 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-15 02:56:00,783 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:56:00,783 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:56:00,784 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:00,784 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-15 02:56:00,784 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-15 02:56:00,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-15 02:56:00,784 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:00,784 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-15 02:56:00,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:56:00,784 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:56:00,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-15 02:56:00,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:56:00,785 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-15 02:56:00,785 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-15 02:56:00,785 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-15 02:56:00,785 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-15 02:56:00,785 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-15 02:56:00,785 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-15 02:56:00,785 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-15 02:56:00,785 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-15 02:56:00,785 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681527356526.20e761db5975fb9082f870bf056bd078. 2023-04-15 02:56:00,785 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681527356526.20e761db5975fb9082f870bf056bd078. started... 2023-04-15 02:56:00,785 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681527356526.20e761db5975fb9082f870bf056bd078. as already flushing 2023-04-15 02:56:00,785 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681527356526.20e761db5975fb9082f870bf056bd078. 2023-04-15 02:56:00,785 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-15 02:56:00,785 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-15 02:56:00,785 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:00,785 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-15 02:56:00,785 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34681,1681527209133' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-15 02:56:00,786 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:00,786 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:56:00,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:00,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:56:00,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:56:00,786 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:56:00,786 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-15 02:56:00,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:56:00,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:56:00,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:56:00,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:00,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:56:00,787 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34681,1681527209133' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-15 02:56:00,787 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-15 02:56:00,787 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@602474c2[Count = 0] remaining members to acquire global barrier 2023-04-15 02:56:00,787 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:56:00,788 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:56:00,788 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:56:00,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:56:00,788 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-15 02:56:00,788 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-15 02:56:00,788 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:00,788 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-15 02:56:00,788 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34681,1681527209133' in zk 2023-04-15 02:56:00,788 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-15 02:56:00,788 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:00,788 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-15 02:56:00,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:00,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:56:00,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:56:00,788 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-15 02:56:00,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:56:00,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:56:00,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:56:00,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:00,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:56:00,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:56:00,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:00,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34681,1681527209133': 2023-04-15 02:56:00,790 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34681,1681527209133' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-15 02:56:00,790 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-15 02:56:00,790 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-15 02:56:00,790 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-15 02:56:00,790 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-15 02:56:00,790 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-15 02:56:00,791 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:56:00,791 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:56:00,791 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:56:00,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:56:00,791 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-15 02:56:00,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:56:00,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:56:00,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:56:00,791 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:00,791 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-15 02:56:00,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-15 02:56:00,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:56:00,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:56:00,792 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:56:00,792 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:56:00,792 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:00,792 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:56:00,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:00,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:56:00,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:56:00,793 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:00,794 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:56:00,794 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:56:00,794 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:00,794 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:56:00,794 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-15 02:56:00,794 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-15 02:56:00,794 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-15 02:56:00,794 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-15 02:56:00,794 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:56:00,794 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:56:00,794 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-15 02:56:00,794 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-15 02:56:00,794 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-15 02:56:00,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-15 02:56:00,794 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-15 02:56:00,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-15 02:56:00,796 DEBUG [Thread-3439] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-15 02:56:00,796 DEBUG [Thread-3439] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-15 02:56:00,799 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=20e761db5975fb9082f870bf056bd078, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:56:00,799 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=20e761db5975fb9082f870bf056bd078, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:56:00,799 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34681] ipc.CallRunner(144): callId: 31 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:48666 deadline: 1681527420796, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=20e761db5975fb9082f870bf056bd078, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:00,799 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34681] ipc.CallRunner(144): callId: 34 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:48664 deadline: 1681527420797, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=20e761db5975fb9082f870bf056bd078, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:00,799 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=20e761db5975fb9082f870bf056bd078, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:56:00,799 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=20e761db5975fb9082f870bf056bd078, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:56:00,799 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34681] ipc.CallRunner(144): callId: 35 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:48648 deadline: 1681527420797, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=20e761db5975fb9082f870bf056bd078, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:00,799 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34681] ipc.CallRunner(144): callId: 33 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:48624 deadline: 1681527420797, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=20e761db5975fb9082f870bf056bd078, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:00,800 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=20e761db5975fb9082f870bf056bd078, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:56:00,800 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34681] ipc.CallRunner(144): callId: 34 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:48586 deadline: 1681527420798, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=20e761db5975fb9082f870bf056bd078, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:00,893 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=64, memsize=24.6 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/20e761db5975fb9082f870bf056bd078/.tmp/A/3e28da5a132b4b5e9abf7f7e92c7bde4 2023-04-15 02:56:00,896 DEBUG [Thread-3439] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-15 02:56:00,897 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-15 02:56:00,898 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-15 02:56:00,899 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-15 02:56:00,899 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-15 02:56:00,899 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-15 02:56:00,899 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-15 02:56:00,899 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-15 02:56:00,899 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-15 02:56:00,899 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:56:00,900 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:56:00,900 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-15 02:56:00,900 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-15 02:56:00,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-15 02:56:00,900 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:00,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:56:00,901 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=64 (bloomFilter=true), to=hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/20e761db5975fb9082f870bf056bd078/.tmp/B/cc033b3c9f6e443b9aa13784606e91be 2023-04-15 02:56:00,901 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:00,901 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-15 02:56:00,901 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:56:00,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-15 02:56:00,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:56:00,901 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-15 02:56:00,901 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-15 02:56:00,901 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-15 02:56:00,901 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-15 02:56:00,901 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-15 02:56:00,902 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-15 02:56:00,902 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-15 02:56:00,902 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-15 02:56:00,902 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681527356526.20e761db5975fb9082f870bf056bd078. 2023-04-15 02:56:00,902 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681527356526.20e761db5975fb9082f870bf056bd078. started... 2023-04-15 02:56:00,902 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681527356526.20e761db5975fb9082f870bf056bd078. as already flushing 2023-04-15 02:56:00,902 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681527356526.20e761db5975fb9082f870bf056bd078. 2023-04-15 02:56:00,902 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-15 02:56:00,902 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-15 02:56:00,902 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:00,902 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-15 02:56:00,902 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34681,1681527209133' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-15 02:56:00,903 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:00,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:00,903 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:56:00,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:56:00,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:56:00,903 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:56:00,903 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-15 02:56:00,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:56:00,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:56:00,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:56:00,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:00,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:56:00,905 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34681,1681527209133' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-15 02:56:00,905 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-15 02:56:00,905 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4e4727ad[Count = 0] remaining members to acquire global barrier 2023-04-15 02:56:00,905 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:56:00,906 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:56:00,906 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:56:00,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:56:00,906 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-15 02:56:00,906 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-15 02:56:00,906 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34681,1681527209133' in zk 2023-04-15 02:56:00,906 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:00,907 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-15 02:56:00,907 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-15 02:56:00,907 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:00,907 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-15 02:56:00,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:00,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:56:00,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:56:00,907 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-15 02:56:00,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:56:00,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:56:00,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:56:00,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:00,909 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=64 (bloomFilter=true), to=hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/20e761db5975fb9082f870bf056bd078/.tmp/C/43db6e1add50473786a2b33dee3985cd 2023-04-15 02:56:00,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:56:00,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:56:00,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:00,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34681,1681527209133': 2023-04-15 02:56:00,910 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34681,1681527209133' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-15 02:56:00,910 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-15 02:56:00,911 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-15 02:56:00,911 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-15 02:56:00,911 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-15 02:56:00,911 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-15 02:56:00,912 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:56:00,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:56:00,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:56:00,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:56:00,912 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:56:00,912 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-15 02:56:00,912 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:56:00,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:56:00,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:56:00,912 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/20e761db5975fb9082f870bf056bd078/.tmp/A/3e28da5a132b4b5e9abf7f7e92c7bde4 as hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/20e761db5975fb9082f870bf056bd078/A/3e28da5a132b4b5e9abf7f7e92c7bde4 2023-04-15 02:56:00,912 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-15 02:56:00,912 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:00,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-15 02:56:00,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:56:00,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:56:00,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:56:00,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:56:00,914 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:00,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:00,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:56:00,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:56:00,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:00,916 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/20e761db5975fb9082f870bf056bd078/A/3e28da5a132b4b5e9abf7f7e92c7bde4, entries=150, sequenceid=64, filesize=25.1 K 2023-04-15 02:56:00,916 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-15 02:56:00,916 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-15 02:56:00,916 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-15 02:56:00,916 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-15 02:56:00,916 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:00,916 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-15 02:56:00,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-15 02:56:00,916 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-15 02:56:00,917 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-15 02:56:00,916 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:56:00,917 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:56:00,917 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:00,917 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:56:00,917 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:56:00,917 DEBUG [Thread-3439] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-15 02:56:00,917 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:56:00,917 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-15 02:56:00,917 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/20e761db5975fb9082f870bf056bd078/.tmp/B/cc033b3c9f6e443b9aa13784606e91be as hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/20e761db5975fb9082f870bf056bd078/B/cc033b3c9f6e443b9aa13784606e91be 2023-04-15 02:56:00,917 DEBUG [Thread-3439] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-15 02:56:00,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-15 02:56:00,919 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/20e761db5975fb9082f870bf056bd078/B/cc033b3c9f6e443b9aa13784606e91be, entries=150, sequenceid=64, filesize=11.6 K 2023-04-15 02:56:00,920 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/20e761db5975fb9082f870bf056bd078/.tmp/C/43db6e1add50473786a2b33dee3985cd as hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/20e761db5975fb9082f870bf056bd078/C/43db6e1add50473786a2b33dee3985cd 2023-04-15 02:56:00,922 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/20e761db5975fb9082f870bf056bd078/C/43db6e1add50473786a2b33dee3985cd, entries=150, sequenceid=64, filesize=11.6 K 2023-04-15 02:56:00,923 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~73.80 KB/75570, heapSize ~219.84 KB/225120, currentSize=107.34 KB/109920 for 20e761db5975fb9082f870bf056bd078 in 444ms, sequenceid=64, compaction requested=true 2023-04-15 02:56:00,923 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 20e761db5975fb9082f870bf056bd078: 2023-04-15 02:56:00,923 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-15 02:56:00,923 DEBUG [RS:0;jenkins-hbase20:34681-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-15 02:56:00,923 DEBUG [RS:0;jenkins-hbase20:34681-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-15 02:56:00,923 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-15 02:56:00,923 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-15 02:56:00,924 DEBUG [RS:0;jenkins-hbase20:34681-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 77238 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-15 02:56:00,924 DEBUG [RS:0;jenkins-hbase20:34681-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 35658 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-15 02:56:00,924 DEBUG [RS:0;jenkins-hbase20:34681-shortCompactions-0] regionserver.HStore(1912): 20e761db5975fb9082f870bf056bd078/A is initiating minor compaction (all files) 2023-04-15 02:56:00,924 DEBUG [RS:0;jenkins-hbase20:34681-longCompactions-0] regionserver.HStore(1912): 20e761db5975fb9082f870bf056bd078/B is initiating minor compaction (all files) 2023-04-15 02:56:00,924 INFO [RS:0;jenkins-hbase20:34681-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 20e761db5975fb9082f870bf056bd078/A in TestAcidGuarantees,,1681527356526.20e761db5975fb9082f870bf056bd078. 2023-04-15 02:56:00,924 INFO [RS:0;jenkins-hbase20:34681-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 20e761db5975fb9082f870bf056bd078/B in TestAcidGuarantees,,1681527356526.20e761db5975fb9082f870bf056bd078. 2023-04-15 02:56:00,924 INFO [RS:0;jenkins-hbase20:34681-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/20e761db5975fb9082f870bf056bd078/A/a714ed51fc7c46378b9e8630cea6f95f, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/20e761db5975fb9082f870bf056bd078/A/78ddfefdb20c4498b0be0f13b8b9bf97, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/20e761db5975fb9082f870bf056bd078/A/3e28da5a132b4b5e9abf7f7e92c7bde4] into tmpdir=hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/20e761db5975fb9082f870bf056bd078/.tmp, totalSize=75.4 K 2023-04-15 02:56:00,924 INFO [RS:0;jenkins-hbase20:34681-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/20e761db5975fb9082f870bf056bd078/B/ce7e96538ce54fc2bed1e90c526b8f0e, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/20e761db5975fb9082f870bf056bd078/B/a78ad28d40df4cb39afa5886b9b0cf03, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/20e761db5975fb9082f870bf056bd078/B/cc033b3c9f6e443b9aa13784606e91be] into tmpdir=hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/20e761db5975fb9082f870bf056bd078/.tmp, totalSize=34.8 K 2023-04-15 02:56:00,924 DEBUG [RS:0;jenkins-hbase20:34681-shortCompactions-0] compactions.Compactor(207): Compacting a714ed51fc7c46378b9e8630cea6f95f, keycount=150, bloomtype=ROW, size=25.1 K, encoding=NONE, compression=NONE, seqNum=16, earliestPutTs=1681527358715 2023-04-15 02:56:00,924 DEBUG [RS:0;jenkins-hbase20:34681-longCompactions-0] compactions.Compactor(207): Compacting ce7e96538ce54fc2bed1e90c526b8f0e, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=16, earliestPutTs=1681527358715 2023-04-15 02:56:00,924 DEBUG [RS:0;jenkins-hbase20:34681-shortCompactions-0] compactions.Compactor(207): Compacting 78ddfefdb20c4498b0be0f13b8b9bf97, keycount=150, bloomtype=ROW, size=25.1 K, encoding=NONE, compression=NONE, seqNum=50, earliestPutTs=1681527358742 2023-04-15 02:56:00,924 DEBUG [RS:0;jenkins-hbase20:34681-longCompactions-0] compactions.Compactor(207): Compacting a78ad28d40df4cb39afa5886b9b0cf03, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=50, earliestPutTs=1681527358742 2023-04-15 02:56:00,925 DEBUG [RS:0;jenkins-hbase20:34681-shortCompactions-0] compactions.Compactor(207): Compacting 3e28da5a132b4b5e9abf7f7e92c7bde4, keycount=150, bloomtype=ROW, size=25.1 K, encoding=NONE, compression=NONE, seqNum=64, earliestPutTs=1681527359363 2023-04-15 02:56:00,925 DEBUG [RS:0;jenkins-hbase20:34681-longCompactions-0] compactions.Compactor(207): Compacting cc033b3c9f6e443b9aa13784606e91be, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=64, earliestPutTs=1681527359363 2023-04-15 02:56:00,934 INFO [RS:0;jenkins-hbase20:34681-longCompactions-0] throttle.PressureAwareThroughputController(145): 20e761db5975fb9082f870bf056bd078#B#compaction#449 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-15 02:56:00,940 INFO [RS:0;jenkins-hbase20:34681-shortCompactions-0] throttle.PressureAwareThroughputController(145): 20e761db5975fb9082f870bf056bd078#A#compaction#450 average throughput is 3.94 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-15 02:56:01,017 DEBUG [Thread-3439] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-15 02:56:01,018 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-15 02:56:01,019 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-15 02:56:01,019 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-15 02:56:01,019 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-15 02:56:01,019 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-15 02:56:01,019 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-15 02:56:01,019 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-15 02:56:01,019 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-15 02:56:01,020 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:56:01,020 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:56:01,020 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:01,020 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-15 02:56:01,020 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-15 02:56:01,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-15 02:56:01,021 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:01,021 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-15 02:56:01,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:56:01,021 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:56:01,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-15 02:56:01,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:56:01,021 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-15 02:56:01,021 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-15 02:56:01,021 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-15 02:56:01,021 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-15 02:56:01,021 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-15 02:56:01,021 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-15 02:56:01,021 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-15 02:56:01,021 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-15 02:56:01,021 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681527356526.20e761db5975fb9082f870bf056bd078. 2023-04-15 02:56:01,022 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681527356526.20e761db5975fb9082f870bf056bd078. started... 2023-04-15 02:56:01,022 INFO [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 20e761db5975fb9082f870bf056bd078 3/3 column families, dataSize=107.34 KB heapSize=319.50 KB 2023-04-15 02:56:01,022 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 20e761db5975fb9082f870bf056bd078, store=A 2023-04-15 02:56:01,022 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-15 02:56:01,022 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 20e761db5975fb9082f870bf056bd078, store=B 2023-04-15 02:56:01,022 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-15 02:56:01,022 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 20e761db5975fb9082f870bf056bd078, store=C 2023-04-15 02:56:01,022 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-15 02:56:01,101 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34681] regionserver.HRegion(9158): Flush requested on 20e761db5975fb9082f870bf056bd078 2023-04-15 02:56:01,102 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681527356526.20e761db5975fb9082f870bf056bd078. as already flushing 2023-04-15 02:56:01,109 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=20e761db5975fb9082f870bf056bd078, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:56:01,109 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34681] ipc.CallRunner(144): callId: 38 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:48648 deadline: 1681527421105, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=20e761db5975fb9082f870bf056bd078, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:01,110 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=20e761db5975fb9082f870bf056bd078, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:56:01,110 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34681] ipc.CallRunner(144): callId: 38 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:48586 deadline: 1681527421107, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=20e761db5975fb9082f870bf056bd078, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:01,111 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=20e761db5975fb9082f870bf056bd078, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:56:01,111 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34681] ipc.CallRunner(144): callId: 38 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:48664 deadline: 1681527421108, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=20e761db5975fb9082f870bf056bd078, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:01,111 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=20e761db5975fb9082f870bf056bd078, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:56:01,111 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34681] ipc.CallRunner(144): callId: 36 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:48666 deadline: 1681527421108, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=20e761db5975fb9082f870bf056bd078, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:01,111 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=20e761db5975fb9082f870bf056bd078, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:56:01,111 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34681] ipc.CallRunner(144): callId: 37 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:48624 deadline: 1681527421109, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=20e761db5975fb9082f870bf056bd078, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:01,211 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=20e761db5975fb9082f870bf056bd078, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:56:01,211 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34681] ipc.CallRunner(144): callId: 40 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:48648 deadline: 1681527421210, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=20e761db5975fb9082f870bf056bd078, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:01,212 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=20e761db5975fb9082f870bf056bd078, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:56:01,212 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34681] ipc.CallRunner(144): callId: 40 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:48586 deadline: 1681527421211, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=20e761db5975fb9082f870bf056bd078, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:01,214 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=20e761db5975fb9082f870bf056bd078, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:56:01,214 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=20e761db5975fb9082f870bf056bd078, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:56:01,214 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34681] ipc.CallRunner(144): callId: 40 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:48664 deadline: 1681527421212, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=20e761db5975fb9082f870bf056bd078, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:01,214 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34681] ipc.CallRunner(144): callId: 38 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:48666 deadline: 1681527421212, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=20e761db5975fb9082f870bf056bd078, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:01,215 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=20e761db5975fb9082f870bf056bd078, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:56:01,215 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34681] ipc.CallRunner(144): callId: 39 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:48624 deadline: 1681527421212, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=20e761db5975fb9082f870bf056bd078, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:01,344 DEBUG [RS:0;jenkins-hbase20:34681-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/20e761db5975fb9082f870bf056bd078/.tmp/B/0969d679f2bb4597bac256d78f824067 as hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/20e761db5975fb9082f870bf056bd078/B/0969d679f2bb4597bac256d78f824067 2023-04-15 02:56:01,348 INFO [RS:0;jenkins-hbase20:34681-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 20e761db5975fb9082f870bf056bd078/B of 20e761db5975fb9082f870bf056bd078 into 0969d679f2bb4597bac256d78f824067(size=11.7 K), total size for store is 11.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-15 02:56:01,348 DEBUG [RS:0;jenkins-hbase20:34681-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 20e761db5975fb9082f870bf056bd078: 2023-04-15 02:56:01,349 INFO [RS:0;jenkins-hbase20:34681-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681527356526.20e761db5975fb9082f870bf056bd078., storeName=20e761db5975fb9082f870bf056bd078/B, priority=13, startTime=1681527360923; duration=0sec 2023-04-15 02:56:01,349 DEBUG [RS:0;jenkins-hbase20:34681-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-15 02:56:01,349 DEBUG [RS:0;jenkins-hbase20:34681-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-15 02:56:01,350 DEBUG [RS:0;jenkins-hbase20:34681-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 35658 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-15 02:56:01,350 DEBUG [RS:0;jenkins-hbase20:34681-longCompactions-0] regionserver.HStore(1912): 20e761db5975fb9082f870bf056bd078/C is initiating minor compaction (all files) 2023-04-15 02:56:01,350 INFO [RS:0;jenkins-hbase20:34681-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 20e761db5975fb9082f870bf056bd078/C in TestAcidGuarantees,,1681527356526.20e761db5975fb9082f870bf056bd078. 2023-04-15 02:56:01,350 INFO [RS:0;jenkins-hbase20:34681-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/20e761db5975fb9082f870bf056bd078/C/bc0770e966664656b1a39dc2b7bc02ad, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/20e761db5975fb9082f870bf056bd078/C/3d55dce025b5415982c8518bb01bd7d4, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/20e761db5975fb9082f870bf056bd078/C/43db6e1add50473786a2b33dee3985cd] into tmpdir=hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/20e761db5975fb9082f870bf056bd078/.tmp, totalSize=34.8 K 2023-04-15 02:56:01,351 DEBUG [RS:0;jenkins-hbase20:34681-longCompactions-0] compactions.Compactor(207): Compacting bc0770e966664656b1a39dc2b7bc02ad, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=16, earliestPutTs=1681527358715 2023-04-15 02:56:01,351 DEBUG [RS:0;jenkins-hbase20:34681-longCompactions-0] compactions.Compactor(207): Compacting 3d55dce025b5415982c8518bb01bd7d4, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=50, earliestPutTs=1681527358742 2023-04-15 02:56:01,351 DEBUG [RS:0;jenkins-hbase20:34681-longCompactions-0] compactions.Compactor(207): Compacting 43db6e1add50473786a2b33dee3985cd, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=64, earliestPutTs=1681527359363 2023-04-15 02:56:01,356 INFO [RS:0;jenkins-hbase20:34681-longCompactions-0] throttle.PressureAwareThroughputController(145): 20e761db5975fb9082f870bf056bd078#C#compaction#452 average throughput is unlimited, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-15 02:56:01,365 DEBUG [RS:0;jenkins-hbase20:34681-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/20e761db5975fb9082f870bf056bd078/.tmp/C/6dd346ee5c4e4a1f8bf4d4dff5b0287c as hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/20e761db5975fb9082f870bf056bd078/C/6dd346ee5c4e4a1f8bf4d4dff5b0287c 2023-04-15 02:56:01,369 INFO [RS:0;jenkins-hbase20:34681-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 20e761db5975fb9082f870bf056bd078/C of 20e761db5975fb9082f870bf056bd078 into 6dd346ee5c4e4a1f8bf4d4dff5b0287c(size=11.7 K), total size for store is 11.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-15 02:56:01,369 DEBUG [RS:0;jenkins-hbase20:34681-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 20e761db5975fb9082f870bf056bd078: 2023-04-15 02:56:01,369 INFO [RS:0;jenkins-hbase20:34681-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681527356526.20e761db5975fb9082f870bf056bd078., storeName=20e761db5975fb9082f870bf056bd078/C, priority=13, startTime=1681527360923; duration=0sec 2023-04-15 02:56:01,369 DEBUG [RS:0;jenkins-hbase20:34681-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-15 02:56:01,413 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=20e761db5975fb9082f870bf056bd078, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:56:01,413 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34681] ipc.CallRunner(144): callId: 42 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:48648 deadline: 1681527421412, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=20e761db5975fb9082f870bf056bd078, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:01,415 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=20e761db5975fb9082f870bf056bd078, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:56:01,415 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34681] ipc.CallRunner(144): callId: 42 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:48586 deadline: 1681527421414, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=20e761db5975fb9082f870bf056bd078, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:01,418 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=20e761db5975fb9082f870bf056bd078, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:56:01,418 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=20e761db5975fb9082f870bf056bd078, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:56:01,418 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34681] ipc.CallRunner(144): callId: 40 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:48666 deadline: 1681527421416, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=20e761db5975fb9082f870bf056bd078, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:01,418 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34681] ipc.CallRunner(144): callId: 42 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:48664 deadline: 1681527421416, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=20e761db5975fb9082f870bf056bd078, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:01,418 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=20e761db5975fb9082f870bf056bd078, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:56:01,418 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34681] ipc.CallRunner(144): callId: 41 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:48624 deadline: 1681527421416, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=20e761db5975fb9082f870bf056bd078, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:01,434 INFO [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023041537ea055f08164e7fab9c6d42d004c4ab to hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023041537ea055f08164e7fab9c6d42d004c4ab 2023-04-15 02:56:01,444 WARN [HBase-Metrics2-1] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties 2023-04-15 02:56:01,714 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=20e761db5975fb9082f870bf056bd078, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:56:01,714 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34681] ipc.CallRunner(144): callId: 44 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:48648 deadline: 1681527421714, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=20e761db5975fb9082f870bf056bd078, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:01,721 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=20e761db5975fb9082f870bf056bd078, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:56:01,721 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34681] ipc.CallRunner(144): callId: 44 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:48586 deadline: 1681527421718, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=20e761db5975fb9082f870bf056bd078, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:01,721 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=20e761db5975fb9082f870bf056bd078, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:56:01,721 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34681] ipc.CallRunner(144): callId: 44 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:48664 deadline: 1681527421719, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=20e761db5975fb9082f870bf056bd078, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:01,722 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=20e761db5975fb9082f870bf056bd078, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:56:01,722 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34681] ipc.CallRunner(144): callId: 42 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:48666 deadline: 1681527421720, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=20e761db5975fb9082f870bf056bd078, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:01,722 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=20e761db5975fb9082f870bf056bd078, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:56:01,722 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34681] ipc.CallRunner(144): callId: 43 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:48624 deadline: 1681527421720, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=20e761db5975fb9082f870bf056bd078, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:01,764 DEBUG [RS:0;jenkins-hbase20:34681-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/20e761db5975fb9082f870bf056bd078/.tmp/A/7f2af62291f44d66a5b3f7a3ff162745 as hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/20e761db5975fb9082f870bf056bd078/A/7f2af62291f44d66a5b3f7a3ff162745 2023-04-15 02:56:01,767 INFO [RS:0;jenkins-hbase20:34681-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 20e761db5975fb9082f870bf056bd078/A of 20e761db5975fb9082f870bf056bd078 into 7f2af62291f44d66a5b3f7a3ff162745(size=25.2 K), total size for store is 25.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-15 02:56:01,768 DEBUG [RS:0;jenkins-hbase20:34681-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 20e761db5975fb9082f870bf056bd078: 2023-04-15 02:56:01,768 INFO [RS:0;jenkins-hbase20:34681-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681527356526.20e761db5975fb9082f870bf056bd078., storeName=20e761db5975fb9082f870bf056bd078/A, priority=13, startTime=1681527360923; duration=0sec 2023-04-15 02:56:01,768 DEBUG [RS:0;jenkins-hbase20:34681-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-15 02:56:01,838 INFO [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=83, memsize=35.8 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/20e761db5975fb9082f870bf056bd078/.tmp/A/682a33ab364b4412a3f05a564e0c8208 2023-04-15 02:56:02,216 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=20e761db5975fb9082f870bf056bd078, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:56:02,216 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34681] ipc.CallRunner(144): callId: 46 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:48648 deadline: 1681527422215, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=20e761db5975fb9082f870bf056bd078, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:02,226 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=20e761db5975fb9082f870bf056bd078, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:56:02,227 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34681] ipc.CallRunner(144): callId: 46 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:48664 deadline: 1681527422226, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=20e761db5975fb9082f870bf056bd078, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:02,227 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=20e761db5975fb9082f870bf056bd078, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:56:02,227 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34681] ipc.CallRunner(144): callId: 45 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:48624 deadline: 1681527422226, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=20e761db5975fb9082f870bf056bd078, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:02,227 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=20e761db5975fb9082f870bf056bd078, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:56:02,227 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34681] ipc.CallRunner(144): callId: 46 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:48586 deadline: 1681527422226, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=20e761db5975fb9082f870bf056bd078, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:02,227 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=20e761db5975fb9082f870bf056bd078, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:56:02,227 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34681] ipc.CallRunner(144): callId: 44 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:48666 deadline: 1681527422226, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=20e761db5975fb9082f870bf056bd078, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:02,246 INFO [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=83 (bloomFilter=true), to=hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/20e761db5975fb9082f870bf056bd078/.tmp/B/0c295b6fa6c34914ba5164667243e923 2023-04-15 02:56:02,653 INFO [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=83 (bloomFilter=true), to=hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/20e761db5975fb9082f870bf056bd078/.tmp/C/06699938be04491a822b2e05301c55a8 2023-04-15 02:56:02,656 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/20e761db5975fb9082f870bf056bd078/.tmp/A/682a33ab364b4412a3f05a564e0c8208 as hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/20e761db5975fb9082f870bf056bd078/A/682a33ab364b4412a3f05a564e0c8208 2023-04-15 02:56:02,659 INFO [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/20e761db5975fb9082f870bf056bd078/A/682a33ab364b4412a3f05a564e0c8208, entries=150, sequenceid=83, filesize=25.1 K 2023-04-15 02:56:02,659 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/20e761db5975fb9082f870bf056bd078/.tmp/B/0c295b6fa6c34914ba5164667243e923 as hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/20e761db5975fb9082f870bf056bd078/B/0c295b6fa6c34914ba5164667243e923 2023-04-15 02:56:02,662 INFO [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/20e761db5975fb9082f870bf056bd078/B/0c295b6fa6c34914ba5164667243e923, entries=150, sequenceid=83, filesize=11.6 K 2023-04-15 02:56:02,662 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/20e761db5975fb9082f870bf056bd078/.tmp/C/06699938be04491a822b2e05301c55a8 as hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/20e761db5975fb9082f870bf056bd078/C/06699938be04491a822b2e05301c55a8 2023-04-15 02:56:02,665 INFO [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/20e761db5975fb9082f870bf056bd078/C/06699938be04491a822b2e05301c55a8, entries=150, sequenceid=83, filesize=11.6 K 2023-04-15 02:56:02,666 INFO [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~107.34 KB/109920, heapSize ~319.45 KB/327120, currentSize=67.09 KB/68700 for 20e761db5975fb9082f870bf056bd078 in 1644ms, sequenceid=83, compaction requested=false 2023-04-15 02:56:02,666 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 20e761db5975fb9082f870bf056bd078: 2023-04-15 02:56:02,666 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681527356526.20e761db5975fb9082f870bf056bd078. 2023-04-15 02:56:02,666 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-15 02:56:02,666 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-15 02:56:02,666 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:02,666 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-15 02:56:02,666 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34681,1681527209133' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-15 02:56:02,667 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:02,667 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:56:02,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:02,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:56:02,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:56:02,667 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:56:02,667 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-15 02:56:02,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:56:02,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:56:02,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:56:02,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:02,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:56:02,668 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34681,1681527209133' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-15 02:56:02,668 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@366ea592[Count = 0] remaining members to acquire global barrier 2023-04-15 02:56:02,668 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-15 02:56:02,668 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:56:02,670 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:56:02,670 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:56:02,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:56:02,670 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-15 02:56:02,670 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:02,670 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-15 02:56:02,670 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-15 02:56:02,670 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34681,1681527209133' in zk 2023-04-15 02:56:02,671 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-15 02:56:02,671 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:02,671 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-15 02:56:02,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:02,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:56:02,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:56:02,671 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-15 02:56:02,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:56:02,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:56:02,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:56:02,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:02,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:56:02,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:56:02,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:02,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34681,1681527209133': 2023-04-15 02:56:02,673 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34681,1681527209133' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-15 02:56:02,673 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-15 02:56:02,673 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-15 02:56:02,673 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-15 02:56:02,673 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-15 02:56:02,673 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-15 02:56:02,674 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:56:02,674 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-15 02:56:02,674 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:56:02,674 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:56:02,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:56:02,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:56:02,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:56:02,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:56:02,674 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-15 02:56:02,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-15 02:56:02,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:56:02,675 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:02,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:56:02,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:56:02,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:56:02,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:56:02,675 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:02,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:02,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:56:02,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:56:02,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:02,677 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:02,677 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:56:02,677 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:56:02,677 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:02,677 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:56:02,677 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-15 02:56:02,677 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-15 02:56:02,677 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-15 02:56:02,677 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-15 02:56:02,678 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-15 02:56:02,678 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-15 02:56:02,677 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:56:02,678 DEBUG [Thread-3439] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-15 02:56:02,678 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:56:02,678 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-15 02:56:02,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-15 02:56:02,678 DEBUG [Thread-3439] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-15 02:56:02,678 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-15 02:56:02,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-15 02:56:02,778 DEBUG [Thread-3439] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-15 02:56:02,778 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-15 02:56:02,780 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-15 02:56:02,781 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-15 02:56:02,781 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-15 02:56:02,781 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-15 02:56:02,781 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-15 02:56:02,781 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-15 02:56:02,781 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-15 02:56:02,781 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:56:02,781 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:56:02,782 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:02,782 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-15 02:56:02,782 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-15 02:56:02,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-15 02:56:02,782 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:02,782 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-15 02:56:02,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:56:02,782 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:56:02,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-15 02:56:02,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:56:02,782 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-15 02:56:02,783 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-15 02:56:02,783 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-15 02:56:02,783 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-15 02:56:02,783 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-15 02:56:02,783 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-15 02:56:02,783 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-15 02:56:02,783 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-15 02:56:02,783 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681527356526.20e761db5975fb9082f870bf056bd078. 2023-04-15 02:56:02,783 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681527356526.20e761db5975fb9082f870bf056bd078. started... 2023-04-15 02:56:02,783 INFO [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 20e761db5975fb9082f870bf056bd078 3/3 column families, dataSize=67.09 KB heapSize=199.97 KB 2023-04-15 02:56:02,783 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 20e761db5975fb9082f870bf056bd078, store=A 2023-04-15 02:56:02,783 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-15 02:56:02,783 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 20e761db5975fb9082f870bf056bd078, store=B 2023-04-15 02:56:02,783 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-15 02:56:02,783 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 20e761db5975fb9082f870bf056bd078, store=C 2023-04-15 02:56:02,783 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-15 02:56:02,810 INFO [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023041581b96046be2248c9a6096a74bbe1191c to hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023041581b96046be2248c9a6096a74bbe1191c 2023-04-15 02:56:03,214 INFO [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=99, memsize=22.4 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/20e761db5975fb9082f870bf056bd078/.tmp/A/61964aa8922847d490712f3d4966e25c 2023-04-15 02:56:03,222 INFO [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=99 (bloomFilter=true), to=hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/20e761db5975fb9082f870bf056bd078/.tmp/B/c030a2631f8f4bc0b69a4c216dc58a93 2023-04-15 02:56:03,228 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34681] regionserver.HRegion(9158): Flush requested on 20e761db5975fb9082f870bf056bd078 2023-04-15 02:56:03,228 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681527356526.20e761db5975fb9082f870bf056bd078. as already flushing 2023-04-15 02:56:03,230 INFO [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=99 (bloomFilter=true), to=hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/20e761db5975fb9082f870bf056bd078/.tmp/C/6c289f99aeaa4124bd3fd318af17610c 2023-04-15 02:56:03,233 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/20e761db5975fb9082f870bf056bd078/.tmp/A/61964aa8922847d490712f3d4966e25c as hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/20e761db5975fb9082f870bf056bd078/A/61964aa8922847d490712f3d4966e25c 2023-04-15 02:56:03,236 INFO [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/20e761db5975fb9082f870bf056bd078/A/61964aa8922847d490712f3d4966e25c, entries=150, sequenceid=99, filesize=25.1 K 2023-04-15 02:56:03,236 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/20e761db5975fb9082f870bf056bd078/.tmp/B/c030a2631f8f4bc0b69a4c216dc58a93 as hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/20e761db5975fb9082f870bf056bd078/B/c030a2631f8f4bc0b69a4c216dc58a93 2023-04-15 02:56:03,242 INFO [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/20e761db5975fb9082f870bf056bd078/B/c030a2631f8f4bc0b69a4c216dc58a93, entries=150, sequenceid=99, filesize=11.6 K 2023-04-15 02:56:03,243 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/20e761db5975fb9082f870bf056bd078/.tmp/C/6c289f99aeaa4124bd3fd318af17610c as hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/20e761db5975fb9082f870bf056bd078/C/6c289f99aeaa4124bd3fd318af17610c 2023-04-15 02:56:03,247 INFO [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/20e761db5975fb9082f870bf056bd078/C/6c289f99aeaa4124bd3fd318af17610c, entries=150, sequenceid=99, filesize=11.6 K 2023-04-15 02:56:03,248 INFO [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~67.09 KB/68700, heapSize ~199.92 KB/204720, currentSize=107.34 KB/109920 for 20e761db5975fb9082f870bf056bd078 in 465ms, sequenceid=99, compaction requested=true 2023-04-15 02:56:03,248 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 20e761db5975fb9082f870bf056bd078: 2023-04-15 02:56:03,248 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681527356526.20e761db5975fb9082f870bf056bd078. 2023-04-15 02:56:03,248 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-15 02:56:03,248 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-15 02:56:03,248 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:03,249 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-15 02:56:03,249 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34681,1681527209133' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-15 02:56:03,249 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:03,249 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:56:03,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:03,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:56:03,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:56:03,250 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:56:03,250 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-15 02:56:03,250 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34681] regionserver.HRegion(9158): Flush requested on 20e761db5975fb9082f870bf056bd078 2023-04-15 02:56:03,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:56:03,250 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 20e761db5975fb9082f870bf056bd078 3/3 column families, dataSize=114.05 KB heapSize=339.42 KB 2023-04-15 02:56:03,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:56:03,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:56:03,251 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 20e761db5975fb9082f870bf056bd078, store=A 2023-04-15 02:56:03,251 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-15 02:56:03,251 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 20e761db5975fb9082f870bf056bd078, store=B 2023-04-15 02:56:03,251 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-15 02:56:03,251 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 20e761db5975fb9082f870bf056bd078, store=C 2023-04-15 02:56:03,251 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-15 02:56:03,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:03,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:56:03,252 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34681,1681527209133' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-15 02:56:03,252 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4a812db3[Count = 0] remaining members to acquire global barrier 2023-04-15 02:56:03,252 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-15 02:56:03,252 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:56:03,253 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:56:03,253 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:56:03,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:56:03,253 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-15 02:56:03,253 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-15 02:56:03,253 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:03,253 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-15 02:56:03,253 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34681,1681527209133' in zk 2023-04-15 02:56:03,254 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:03,254 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-15 02:56:03,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:03,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:56:03,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:56:03,254 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-15 02:56:03,254 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-15 02:56:03,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:56:03,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:56:03,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:56:03,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:03,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:56:03,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:56:03,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:03,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34681,1681527209133': 2023-04-15 02:56:03,255 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34681,1681527209133' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-15 02:56:03,255 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-15 02:56:03,255 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-15 02:56:03,256 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-15 02:56:03,256 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-15 02:56:03,256 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-15 02:56:03,256 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:56:03,256 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:56:03,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:56:03,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:56:03,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:56:03,256 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:56:03,256 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-15 02:56:03,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:56:03,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:56:03,257 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:03,257 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-15 02:56:03,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-15 02:56:03,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:56:03,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:56:03,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:56:03,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:56:03,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:03,258 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:03,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:56:03,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:56:03,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:03,259 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:03,259 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:56:03,259 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-15 02:56:03,259 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-15 02:56:03,259 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-15 02:56:03,259 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:56:03,259 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-15 02:56:03,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-15 02:56:03,259 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-15 02:56:03,259 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-15 02:56:03,259 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-15 02:56:03,259 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-15 02:56:03,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-15 02:56:03,259 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:03,259 DEBUG [Thread-3439] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-15 02:56:03,260 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:56:03,260 DEBUG [Thread-3439] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-15 02:56:03,260 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:56:03,260 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:56:03,280 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=20e761db5975fb9082f870bf056bd078, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:56:03,280 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34681] ipc.CallRunner(144): callId: 49 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:48666 deadline: 1681527423277, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=20e761db5975fb9082f870bf056bd078, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:03,282 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=20e761db5975fb9082f870bf056bd078, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:56:03,282 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34681] ipc.CallRunner(144): callId: 52 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:48624 deadline: 1681527423278, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=20e761db5975fb9082f870bf056bd078, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:03,282 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=20e761db5975fb9082f870bf056bd078, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:56:03,282 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34681] ipc.CallRunner(144): callId: 54 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:48586 deadline: 1681527423279, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=20e761db5975fb9082f870bf056bd078, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:03,282 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=20e761db5975fb9082f870bf056bd078, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:56:03,283 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34681] ipc.CallRunner(144): callId: 52 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:48664 deadline: 1681527423280, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=20e761db5975fb9082f870bf056bd078, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:03,283 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=20e761db5975fb9082f870bf056bd078, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:56:03,283 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34681] ipc.CallRunner(144): callId: 56 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:48648 deadline: 1681527423280, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=20e761db5975fb9082f870bf056bd078, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:03,360 DEBUG [Thread-3439] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-15 02:56:03,360 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-15 02:56:03,361 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-15 02:56:03,362 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-15 02:56:03,362 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-15 02:56:03,362 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-15 02:56:03,362 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-15 02:56:03,362 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-15 02:56:03,362 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-15 02:56:03,363 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:56:03,363 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:56:03,364 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:03,364 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-15 02:56:03,364 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-15 02:56:03,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-15 02:56:03,364 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:03,364 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-15 02:56:03,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:56:03,364 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:56:03,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-15 02:56:03,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:56:03,365 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-15 02:56:03,365 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-15 02:56:03,365 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-15 02:56:03,365 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-15 02:56:03,365 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-15 02:56:03,365 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-15 02:56:03,365 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-15 02:56:03,365 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-15 02:56:03,365 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681527356526.20e761db5975fb9082f870bf056bd078. 2023-04-15 02:56:03,365 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681527356526.20e761db5975fb9082f870bf056bd078. started... 2023-04-15 02:56:03,365 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681527356526.20e761db5975fb9082f870bf056bd078. as already flushing 2023-04-15 02:56:03,365 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681527356526.20e761db5975fb9082f870bf056bd078. 2023-04-15 02:56:03,365 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-15 02:56:03,365 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-15 02:56:03,365 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:03,365 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-15 02:56:03,365 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34681,1681527209133' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-15 02:56:03,366 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:03,366 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:56:03,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:03,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:56:03,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:56:03,366 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:56:03,366 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-15 02:56:03,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:56:03,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:56:03,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:56:03,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:03,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:56:03,367 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34681,1681527209133' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-15 02:56:03,367 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-15 02:56:03,367 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@940d245[Count = 0] remaining members to acquire global barrier 2023-04-15 02:56:03,367 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:56:03,368 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:56:03,368 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:03,368 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-15 02:56:03,368 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:56:03,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:56:03,368 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-15 02:56:03,368 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-15 02:56:03,368 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34681,1681527209133' in zk 2023-04-15 02:56:03,369 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:03,369 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-15 02:56:03,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:03,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:56:03,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:56:03,369 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-15 02:56:03,369 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-15 02:56:03,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:56:03,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:56:03,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:56:03,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:03,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:56:03,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:56:03,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:03,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34681,1681527209133': 2023-04-15 02:56:03,371 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34681,1681527209133' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-15 02:56:03,371 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-15 02:56:03,371 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-15 02:56:03,371 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-15 02:56:03,371 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-15 02:56:03,371 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-15 02:56:03,372 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:56:03,372 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:56:03,372 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-15 02:56:03,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:56:03,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:56:03,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:56:03,372 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:56:03,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:56:03,372 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:03,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:56:03,372 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-15 02:56:03,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-15 02:56:03,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:56:03,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:56:03,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:56:03,373 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:03,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:56:03,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:03,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:56:03,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:56:03,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:03,375 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-15 02:56:03,375 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-15 02:56:03,375 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-15 02:56:03,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-15 02:56:03,375 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-15 02:56:03,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-15 02:56:03,375 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:03,375 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:56:03,375 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:56:03,375 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-15 02:56:03,375 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:03,375 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-15 02:56:03,375 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-15 02:56:03,375 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:56:03,375 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:56:03,375 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:56:03,375 DEBUG [Thread-3439] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-15 02:56:03,375 DEBUG [Thread-3439] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-15 02:56:03,375 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-15 02:56:03,382 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=20e761db5975fb9082f870bf056bd078, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:56:03,383 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34681] ipc.CallRunner(144): callId: 51 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:48666 deadline: 1681527423381, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=20e761db5975fb9082f870bf056bd078, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:03,384 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=20e761db5975fb9082f870bf056bd078, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:56:03,384 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34681] ipc.CallRunner(144): callId: 54 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:48624 deadline: 1681527423382, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=20e761db5975fb9082f870bf056bd078, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:03,384 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=20e761db5975fb9082f870bf056bd078, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:56:03,384 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34681] ipc.CallRunner(144): callId: 56 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:48586 deadline: 1681527423383, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=20e761db5975fb9082f870bf056bd078, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:03,385 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=20e761db5975fb9082f870bf056bd078, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:56:03,385 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34681] ipc.CallRunner(144): callId: 54 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:48664 deadline: 1681527423383, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=20e761db5975fb9082f870bf056bd078, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:03,385 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=20e761db5975fb9082f870bf056bd078, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:56:03,385 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34681] ipc.CallRunner(144): callId: 58 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:48648 deadline: 1681527423383, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=20e761db5975fb9082f870bf056bd078, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:03,475 DEBUG [Thread-3439] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-15 02:56:03,476 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-15 02:56:03,477 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-15 02:56:03,478 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-15 02:56:03,478 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-15 02:56:03,478 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-15 02:56:03,478 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-15 02:56:03,478 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-15 02:56:03,478 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-15 02:56:03,478 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:56:03,478 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:56:03,479 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-15 02:56:03,479 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:03,479 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-15 02:56:03,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-15 02:56:03,479 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:03,479 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-15 02:56:03,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:56:03,480 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:56:03,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-15 02:56:03,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:56:03,480 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-15 02:56:03,480 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-15 02:56:03,480 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-15 02:56:03,480 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-15 02:56:03,480 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-15 02:56:03,480 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-15 02:56:03,480 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-15 02:56:03,480 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-15 02:56:03,480 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681527356526.20e761db5975fb9082f870bf056bd078. 2023-04-15 02:56:03,481 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681527356526.20e761db5975fb9082f870bf056bd078. started... 2023-04-15 02:56:03,481 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681527356526.20e761db5975fb9082f870bf056bd078. as already flushing 2023-04-15 02:56:03,481 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681527356526.20e761db5975fb9082f870bf056bd078. 2023-04-15 02:56:03,481 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-15 02:56:03,481 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-15 02:56:03,481 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:03,481 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-15 02:56:03,481 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34681,1681527209133' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-15 02:56:03,482 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:03,482 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:56:03,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:03,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:56:03,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:56:03,482 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:56:03,482 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-15 02:56:03,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:56:03,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:56:03,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:56:03,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:03,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:56:03,483 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34681,1681527209133' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-15 02:56:03,483 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7e4df68f[Count = 0] remaining members to acquire global barrier 2023-04-15 02:56:03,483 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-15 02:56:03,483 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:56:03,483 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:56:03,483 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:56:03,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:56:03,484 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-15 02:56:03,484 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-15 02:56:03,484 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:03,484 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-15 02:56:03,484 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34681,1681527209133' in zk 2023-04-15 02:56:03,485 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-15 02:56:03,485 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-15 02:56:03,485 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:03,485 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-15 02:56:03,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:03,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:56:03,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:56:03,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:56:03,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:56:03,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:56:03,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:03,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:56:03,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:56:03,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:03,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34681,1681527209133': 2023-04-15 02:56:03,487 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34681,1681527209133' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-15 02:56:03,487 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-15 02:56:03,487 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-15 02:56:03,487 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-15 02:56:03,487 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-15 02:56:03,487 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-15 02:56:03,488 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:56:03,488 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:56:03,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:56:03,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:56:03,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:56:03,488 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-15 02:56:03,488 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:56:03,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:56:03,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:56:03,488 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-15 02:56:03,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-15 02:56:03,489 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:03,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:56:03,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:56:03,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:56:03,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:56:03,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:03,490 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:03,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:56:03,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:56:03,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:03,491 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-15 02:56:03,491 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:03,491 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-15 02:56:03,491 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-15 02:56:03,491 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-15 02:56:03,491 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-15 02:56:03,491 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-15 02:56:03,491 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:56:03,491 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:56:03,491 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:03,491 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:56:03,492 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:56:03,492 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:56:03,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-15 02:56:03,491 DEBUG [Thread-3439] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-15 02:56:03,492 DEBUG [Thread-3439] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-15 02:56:03,492 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-15 02:56:03,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-15 02:56:03,585 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=20e761db5975fb9082f870bf056bd078, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:56:03,586 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34681] ipc.CallRunner(144): callId: 53 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:48666 deadline: 1681527423583, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=20e761db5975fb9082f870bf056bd078, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:03,586 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=20e761db5975fb9082f870bf056bd078, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:56:03,586 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34681] ipc.CallRunner(144): callId: 56 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:48624 deadline: 1681527423585, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=20e761db5975fb9082f870bf056bd078, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:03,587 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=20e761db5975fb9082f870bf056bd078, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:56:03,587 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34681] ipc.CallRunner(144): callId: 58 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:48586 deadline: 1681527423585, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=20e761db5975fb9082f870bf056bd078, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:03,587 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=20e761db5975fb9082f870bf056bd078, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:56:03,587 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34681] ipc.CallRunner(144): callId: 56 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:48664 deadline: 1681527423585, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=20e761db5975fb9082f870bf056bd078, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:03,587 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=20e761db5975fb9082f870bf056bd078, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:56:03,587 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34681] ipc.CallRunner(144): callId: 60 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:48648 deadline: 1681527423586, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=20e761db5975fb9082f870bf056bd078, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:03,592 DEBUG [Thread-3439] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-15 02:56:03,592 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-15 02:56:03,593 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-15 02:56:03,594 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-15 02:56:03,594 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-15 02:56:03,594 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-15 02:56:03,595 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-15 02:56:03,595 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-15 02:56:03,595 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-15 02:56:03,595 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:56:03,595 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:56:03,596 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-15 02:56:03,596 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:03,596 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-15 02:56:03,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-15 02:56:03,596 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:03,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:56:03,596 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-15 02:56:03,596 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:56:03,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-15 02:56:03,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:56:03,597 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-15 02:56:03,597 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-15 02:56:03,597 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-15 02:56:03,597 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-15 02:56:03,597 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-15 02:56:03,597 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-15 02:56:03,597 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-15 02:56:03,597 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-15 02:56:03,597 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681527356526.20e761db5975fb9082f870bf056bd078. 2023-04-15 02:56:03,597 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681527356526.20e761db5975fb9082f870bf056bd078. started... 2023-04-15 02:56:03,597 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681527356526.20e761db5975fb9082f870bf056bd078. as already flushing 2023-04-15 02:56:03,597 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681527356526.20e761db5975fb9082f870bf056bd078. 2023-04-15 02:56:03,597 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-15 02:56:03,597 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-15 02:56:03,597 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:03,597 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-15 02:56:03,597 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34681,1681527209133' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-15 02:56:03,598 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:03,598 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:56:03,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:03,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:56:03,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:56:03,598 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:56:03,598 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-15 02:56:03,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:56:03,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:56:03,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:56:03,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:03,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:56:03,600 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34681,1681527209133' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-15 02:56:03,600 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-15 02:56:03,600 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3d4c9ee1[Count = 0] remaining members to acquire global barrier 2023-04-15 02:56:03,600 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:56:03,600 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:56:03,600 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:56:03,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:56:03,600 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-15 02:56:03,600 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-15 02:56:03,600 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:03,600 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-15 02:56:03,600 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34681,1681527209133' in zk 2023-04-15 02:56:03,601 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:03,601 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-15 02:56:03,601 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-15 02:56:03,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:03,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:56:03,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:56:03,601 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-15 02:56:03,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:56:03,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:56:03,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:56:03,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:03,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:56:03,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:56:03,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:03,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34681,1681527209133': 2023-04-15 02:56:03,603 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34681,1681527209133' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-15 02:56:03,603 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-15 02:56:03,603 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-15 02:56:03,603 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-15 02:56:03,603 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-15 02:56:03,603 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-15 02:56:03,604 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:56:03,604 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:56:03,604 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:56:03,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:56:03,604 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-15 02:56:03,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:56:03,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:56:03,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:56:03,604 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-15 02:56:03,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-15 02:56:03,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:56:03,604 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:03,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:56:03,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:56:03,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:56:03,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:56:03,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:03,605 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:03,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:56:03,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:56:03,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:03,607 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-15 02:56:03,607 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:03,607 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-15 02:56:03,607 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-15 02:56:03,607 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-15 02:56:03,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-15 02:56:03,607 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-15 02:56:03,607 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-15 02:56:03,607 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:56:03,607 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:56:03,607 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:03,607 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-15 02:56:03,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-15 02:56:03,607 DEBUG [Thread-3439] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-15 02:56:03,607 DEBUG [Thread-3439] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-15 02:56:03,607 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:56:03,607 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:56:03,607 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:56:03,671 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230415614efc38fedf4a8ca58f0196c5d5df6b to hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230415614efc38fedf4a8ca58f0196c5d5df6b 2023-04-15 02:56:03,707 DEBUG [Thread-3439] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-15 02:56:03,708 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-15 02:56:03,709 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-15 02:56:03,710 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-15 02:56:03,710 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-15 02:56:03,710 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-15 02:56:03,710 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-15 02:56:03,710 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-15 02:56:03,710 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-15 02:56:03,711 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:56:03,711 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:56:03,711 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:03,711 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-15 02:56:03,711 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-15 02:56:03,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-15 02:56:03,711 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:03,711 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-15 02:56:03,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:56:03,712 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:56:03,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-15 02:56:03,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:56:03,712 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-15 02:56:03,712 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-15 02:56:03,712 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-15 02:56:03,712 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-15 02:56:03,712 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-15 02:56:03,712 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-15 02:56:03,712 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-15 02:56:03,713 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-15 02:56:03,713 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681527356526.20e761db5975fb9082f870bf056bd078. 2023-04-15 02:56:03,713 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681527356526.20e761db5975fb9082f870bf056bd078. started... 2023-04-15 02:56:03,713 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681527356526.20e761db5975fb9082f870bf056bd078. as already flushing 2023-04-15 02:56:03,713 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681527356526.20e761db5975fb9082f870bf056bd078. 2023-04-15 02:56:03,713 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-15 02:56:03,713 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-15 02:56:03,713 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:03,713 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-15 02:56:03,713 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34681,1681527209133' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-15 02:56:03,714 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:03,714 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:56:03,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:03,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:56:03,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:56:03,714 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:56:03,714 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-15 02:56:03,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:56:03,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:56:03,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:56:03,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:03,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:56:03,715 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34681,1681527209133' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-15 02:56:03,715 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-15 02:56:03,715 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7e63d04e[Count = 0] remaining members to acquire global barrier 2023-04-15 02:56:03,716 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:56:03,716 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:56:03,716 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:56:03,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:56:03,716 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-15 02:56:03,716 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-15 02:56:03,716 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:03,716 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-15 02:56:03,716 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34681,1681527209133' in zk 2023-04-15 02:56:03,717 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:03,717 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-15 02:56:03,717 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-15 02:56:03,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:03,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:56:03,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:56:03,717 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-15 02:56:03,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:56:03,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:56:03,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:56:03,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:03,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:56:03,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:56:03,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:03,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34681,1681527209133': 2023-04-15 02:56:03,719 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34681,1681527209133' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-15 02:56:03,719 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-15 02:56:03,719 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-15 02:56:03,719 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-15 02:56:03,719 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-15 02:56:03,719 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-15 02:56:03,720 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:56:03,720 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:56:03,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:56:03,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:56:03,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:56:03,720 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:56:03,720 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-15 02:56:03,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:56:03,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:56:03,721 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:03,721 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-15 02:56:03,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-15 02:56:03,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:56:03,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:56:03,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:56:03,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:56:03,722 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:03,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:03,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:56:03,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:56:03,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:03,723 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:03,723 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:56:03,723 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-15 02:56:03,723 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-15 02:56:03,723 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-15 02:56:03,723 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-15 02:56:03,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-15 02:56:03,723 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-15 02:56:03,723 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-15 02:56:03,723 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-15 02:56:03,723 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:56:03,724 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:03,724 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:56:03,724 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:56:03,724 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:56:03,724 DEBUG [Thread-3439] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-15 02:56:03,724 DEBUG [Thread-3439] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-15 02:56:03,724 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-15 02:56:03,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-15 02:56:03,824 DEBUG [Thread-3439] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-15 02:56:03,824 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-15 02:56:03,825 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-15 02:56:03,826 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-15 02:56:03,826 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-15 02:56:03,826 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-15 02:56:03,826 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-15 02:56:03,826 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-15 02:56:03,826 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-15 02:56:03,827 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:56:03,827 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:56:03,827 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-15 02:56:03,827 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:03,827 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-15 02:56:03,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-15 02:56:03,828 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:03,828 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-15 02:56:03,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:56:03,828 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:56:03,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-15 02:56:03,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:56:03,828 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-15 02:56:03,828 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-15 02:56:03,828 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-15 02:56:03,828 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-15 02:56:03,828 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-15 02:56:03,829 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-15 02:56:03,829 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-15 02:56:03,829 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681527356526.20e761db5975fb9082f870bf056bd078. 2023-04-15 02:56:03,829 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681527356526.20e761db5975fb9082f870bf056bd078. started... 2023-04-15 02:56:03,829 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681527356526.20e761db5975fb9082f870bf056bd078. as already flushing 2023-04-15 02:56:03,829 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681527356526.20e761db5975fb9082f870bf056bd078. 2023-04-15 02:56:03,829 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-15 02:56:03,829 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-15 02:56:03,829 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:03,829 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-15 02:56:03,829 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34681,1681527209133' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-15 02:56:03,830 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:03,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:03,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:56:03,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:56:03,830 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:56:03,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:56:03,830 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:56:03,830 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-15 02:56:03,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:56:03,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:56:03,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:03,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:56:03,831 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34681,1681527209133' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-15 02:56:03,831 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@713ef73d[Count = 0] remaining members to acquire global barrier 2023-04-15 02:56:03,831 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-15 02:56:03,831 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:56:03,831 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:56:03,832 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:56:03,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:56:03,832 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-15 02:56:03,832 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-15 02:56:03,832 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34681,1681527209133' in zk 2023-04-15 02:56:03,832 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:03,832 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-15 02:56:03,832 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:03,832 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-15 02:56:03,832 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-15 02:56:03,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:03,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:56:03,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:56:03,832 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-15 02:56:03,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:56:03,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:56:03,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:56:03,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:03,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:56:03,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:56:03,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:03,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34681,1681527209133': 2023-04-15 02:56:03,834 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34681,1681527209133' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-15 02:56:03,835 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-15 02:56:03,835 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-15 02:56:03,835 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-15 02:56:03,835 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-15 02:56:03,835 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-15 02:56:03,835 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:56:03,835 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:56:03,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:56:03,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:56:03,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:56:03,835 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:56:03,835 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-15 02:56:03,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:56:03,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:56:03,836 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-15 02:56:03,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-15 02:56:03,836 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:03,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:56:03,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:56:03,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:56:03,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:56:03,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:03,837 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:03,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:56:03,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:56:03,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:03,839 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:03,839 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:56:03,839 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:56:03,839 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-15 02:56:03,839 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:03,839 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-15 02:56:03,839 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-15 02:56:03,839 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:56:03,839 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:56:03,839 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:56:03,839 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-15 02:56:03,839 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-15 02:56:03,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-15 02:56:03,839 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-15 02:56:03,839 DEBUG [Thread-3439] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-15 02:56:03,839 DEBUG [Thread-3439] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-15 02:56:03,840 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-15 02:56:03,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-15 02:56:03,888 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=20e761db5975fb9082f870bf056bd078, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:56:03,888 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34681] ipc.CallRunner(144): callId: 55 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:48666 deadline: 1681527423886, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=20e761db5975fb9082f870bf056bd078, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:03,889 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=20e761db5975fb9082f870bf056bd078, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:56:03,889 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34681] ipc.CallRunner(144): callId: 58 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:48624 deadline: 1681527423887, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=20e761db5975fb9082f870bf056bd078, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:03,889 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=20e761db5975fb9082f870bf056bd078, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:56:03,889 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34681] ipc.CallRunner(144): callId: 60 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:48586 deadline: 1681527423888, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=20e761db5975fb9082f870bf056bd078, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:03,890 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=20e761db5975fb9082f870bf056bd078, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:56:03,890 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34681] ipc.CallRunner(144): callId: 58 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:48664 deadline: 1681527423888, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=20e761db5975fb9082f870bf056bd078, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:03,891 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=20e761db5975fb9082f870bf056bd078, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:56:03,891 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34681] ipc.CallRunner(144): callId: 62 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:48648 deadline: 1681527423890, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=20e761db5975fb9082f870bf056bd078, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:03,939 DEBUG [Thread-3439] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-15 02:56:03,940 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-15 02:56:03,941 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-15 02:56:03,942 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-15 02:56:03,942 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-15 02:56:03,942 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-15 02:56:03,942 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-15 02:56:03,942 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-15 02:56:03,942 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-15 02:56:03,943 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:56:03,943 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:56:03,943 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-15 02:56:03,943 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:03,943 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-15 02:56:03,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-15 02:56:03,944 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:03,944 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-15 02:56:03,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:56:03,944 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:56:03,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-15 02:56:03,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:56:03,944 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-15 02:56:03,944 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-15 02:56:03,944 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-15 02:56:03,944 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-15 02:56:03,945 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-15 02:56:03,945 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-15 02:56:03,945 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-15 02:56:03,945 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-15 02:56:03,945 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681527356526.20e761db5975fb9082f870bf056bd078. 2023-04-15 02:56:03,945 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681527356526.20e761db5975fb9082f870bf056bd078. started... 2023-04-15 02:56:03,945 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681527356526.20e761db5975fb9082f870bf056bd078. as already flushing 2023-04-15 02:56:03,945 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681527356526.20e761db5975fb9082f870bf056bd078. 2023-04-15 02:56:03,945 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-15 02:56:03,945 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-15 02:56:03,945 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:03,945 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-15 02:56:03,945 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34681,1681527209133' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-15 02:56:03,946 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:56:03,946 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:03,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:03,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:56:03,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:56:03,946 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:56:03,946 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-15 02:56:03,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:56:03,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:56:03,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:56:03,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:03,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:56:03,947 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34681,1681527209133' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-15 02:56:03,947 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-15 02:56:03,947 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@60569e1b[Count = 0] remaining members to acquire global barrier 2023-04-15 02:56:03,947 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:56:03,948 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:56:03,948 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:56:03,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:56:03,948 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-15 02:56:03,948 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-15 02:56:03,948 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:03,948 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-15 02:56:03,948 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34681,1681527209133' in zk 2023-04-15 02:56:03,949 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:03,949 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-15 02:56:03,949 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-15 02:56:03,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:03,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:56:03,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:56:03,949 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-15 02:56:03,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:56:03,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:56:03,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:56:03,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:03,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:56:03,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:56:03,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:03,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34681,1681527209133': 2023-04-15 02:56:03,950 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34681,1681527209133' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-15 02:56:03,950 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-15 02:56:03,951 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-15 02:56:03,951 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-15 02:56:03,951 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-15 02:56:03,951 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-15 02:56:03,951 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:56:03,952 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:56:03,952 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-15 02:56:03,952 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:56:03,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:56:03,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:56:03,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:56:03,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:56:03,952 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-15 02:56:03,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-15 02:56:03,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:56:03,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:56:03,952 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:03,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:56:03,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:56:03,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:56:03,953 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:03,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:03,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:56:03,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:56:03,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:03,955 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:03,955 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:56:03,955 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-15 02:56:03,955 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:56:03,955 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-15 02:56:03,955 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-15 02:56:03,955 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-15 02:56:03,955 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-15 02:56:03,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-15 02:56:03,955 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-15 02:56:03,955 DEBUG [Thread-3439] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-15 02:56:03,955 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-15 02:56:03,955 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:03,955 DEBUG [Thread-3439] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-15 02:56:03,955 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-15 02:56:03,955 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:56:03,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-15 02:56:03,955 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:56:03,955 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:56:04,055 DEBUG [Thread-3439] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-15 02:56:04,056 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-15 02:56:04,057 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-15 02:56:04,058 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-15 02:56:04,058 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-15 02:56:04,058 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-15 02:56:04,058 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-15 02:56:04,058 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-15 02:56:04,058 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-15 02:56:04,058 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:56:04,058 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:56:04,059 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-15 02:56:04,059 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-15 02:56:04,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-15 02:56:04,059 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:04,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:56:04,059 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:04,059 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-15 02:56:04,059 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:56:04,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-15 02:56:04,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:56:04,060 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-15 02:56:04,060 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-15 02:56:04,060 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-15 02:56:04,060 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-15 02:56:04,060 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-15 02:56:04,060 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-15 02:56:04,060 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-15 02:56:04,060 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-15 02:56:04,060 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681527356526.20e761db5975fb9082f870bf056bd078. 2023-04-15 02:56:04,060 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681527356526.20e761db5975fb9082f870bf056bd078. started... 2023-04-15 02:56:04,060 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681527356526.20e761db5975fb9082f870bf056bd078. as already flushing 2023-04-15 02:56:04,060 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681527356526.20e761db5975fb9082f870bf056bd078. 2023-04-15 02:56:04,060 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-15 02:56:04,060 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-15 02:56:04,061 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:04,061 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-15 02:56:04,061 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34681,1681527209133' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-15 02:56:04,061 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:04,061 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:56:04,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:04,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:56:04,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:56:04,062 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:56:04,062 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-15 02:56:04,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:56:04,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:56:04,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:56:04,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:04,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:56:04,063 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34681,1681527209133' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-15 02:56:04,063 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-15 02:56:04,063 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3934bf66[Count = 0] remaining members to acquire global barrier 2023-04-15 02:56:04,063 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:56:04,063 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:56:04,063 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:56:04,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:56:04,063 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-15 02:56:04,063 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:04,063 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-15 02:56:04,063 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-15 02:56:04,063 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34681,1681527209133' in zk 2023-04-15 02:56:04,064 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:04,064 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-15 02:56:04,064 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-15 02:56:04,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:04,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:56:04,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:56:04,064 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-15 02:56:04,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:56:04,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:56:04,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:56:04,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:04,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:56:04,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:56:04,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:04,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34681,1681527209133': 2023-04-15 02:56:04,066 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34681,1681527209133' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-15 02:56:04,066 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-15 02:56:04,066 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-15 02:56:04,066 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-15 02:56:04,066 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-15 02:56:04,066 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-15 02:56:04,067 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:56:04,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:56:04,067 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:56:04,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:56:04,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:56:04,067 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-15 02:56:04,067 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:56:04,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:56:04,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:56:04,067 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:04,067 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-15 02:56:04,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-15 02:56:04,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:56:04,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:56:04,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:56:04,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:56:04,068 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:04,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:04,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:56:04,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:56:04,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:04,069 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:04,069 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:56:04,069 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:56:04,069 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:04,069 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:56:04,069 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-15 02:56:04,069 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-15 02:56:04,069 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-15 02:56:04,069 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-15 02:56:04,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-15 02:56:04,069 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-15 02:56:04,069 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-15 02:56:04,069 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:56:04,069 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-15 02:56:04,070 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:56:04,070 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-15 02:56:04,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-15 02:56:04,070 DEBUG [Thread-3439] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-15 02:56:04,070 DEBUG [Thread-3439] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-15 02:56:04,076 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=120, memsize=40.3 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/20e761db5975fb9082f870bf056bd078/.tmp/A/0312c634961f48589caed89d0a1783c0 2023-04-15 02:56:04,170 DEBUG [Thread-3439] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-15 02:56:04,171 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-15 02:56:04,172 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-15 02:56:04,172 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-15 02:56:04,172 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-15 02:56:04,172 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-15 02:56:04,172 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-15 02:56:04,173 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-15 02:56:04,173 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-15 02:56:04,173 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:56:04,173 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:56:04,173 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:04,173 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-15 02:56:04,173 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-15 02:56:04,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-15 02:56:04,174 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:04,174 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-15 02:56:04,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:56:04,174 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:56:04,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-15 02:56:04,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:56:04,174 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-15 02:56:04,176 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-15 02:56:04,176 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-15 02:56:04,176 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-15 02:56:04,176 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-15 02:56:04,176 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-15 02:56:04,176 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-15 02:56:04,176 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-15 02:56:04,176 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681527356526.20e761db5975fb9082f870bf056bd078. 2023-04-15 02:56:04,176 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681527356526.20e761db5975fb9082f870bf056bd078. started... 2023-04-15 02:56:04,176 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681527356526.20e761db5975fb9082f870bf056bd078. as already flushing 2023-04-15 02:56:04,176 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681527356526.20e761db5975fb9082f870bf056bd078. 2023-04-15 02:56:04,176 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-15 02:56:04,176 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-15 02:56:04,176 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:04,176 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-15 02:56:04,176 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34681,1681527209133' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-15 02:56:04,177 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:04,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:04,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:56:04,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:56:04,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:56:04,179 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:56:04,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:56:04,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:56:04,179 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:56:04,179 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-15 02:56:04,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:04,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:56:04,180 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34681,1681527209133' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-15 02:56:04,180 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@59b5e9ae[Count = 0] remaining members to acquire global barrier 2023-04-15 02:56:04,180 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-15 02:56:04,180 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:56:04,180 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:56:04,180 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:56:04,180 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:04,180 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-15 02:56:04,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:56:04,181 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-15 02:56:04,181 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-15 02:56:04,181 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34681,1681527209133' in zk 2023-04-15 02:56:04,181 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-15 02:56:04,181 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-15 02:56:04,181 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:04,181 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-15 02:56:04,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:04,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:56:04,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:56:04,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:56:04,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:56:04,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:56:04,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:04,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:56:04,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:56:04,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:04,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34681,1681527209133': 2023-04-15 02:56:04,183 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34681,1681527209133' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-15 02:56:04,183 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-15 02:56:04,184 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-15 02:56:04,184 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-15 02:56:04,184 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-15 02:56:04,184 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-15 02:56:04,185 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:56:04,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:56:04,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:56:04,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:56:04,185 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:56:04,185 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-15 02:56:04,185 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:56:04,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:56:04,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:56:04,185 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:04,185 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-15 02:56:04,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-15 02:56:04,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:56:04,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:56:04,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:56:04,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:56:04,186 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:04,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:04,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:56:04,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:56:04,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:04,188 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-15 02:56:04,188 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:04,188 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:56:04,188 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-15 02:56:04,188 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-15 02:56:04,188 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-15 02:56:04,188 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-15 02:56:04,188 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-15 02:56:04,188 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-15 02:56:04,188 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:56:04,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-15 02:56:04,189 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:04,189 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:56:04,189 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:56:04,189 DEBUG [Thread-3439] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-15 02:56:04,189 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:56:04,189 DEBUG [Thread-3439] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-15 02:56:04,189 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-15 02:56:04,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-15 02:56:04,289 DEBUG [Thread-3439] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-15 02:56:04,289 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-15 02:56:04,290 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-15 02:56:04,291 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-15 02:56:04,291 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-15 02:56:04,291 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-15 02:56:04,291 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-15 02:56:04,291 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-15 02:56:04,291 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-15 02:56:04,292 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:56:04,292 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:56:04,312 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:04,312 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-15 02:56:04,312 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-15 02:56:04,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-15 02:56:04,312 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:04,312 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-15 02:56:04,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:56:04,313 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:56:04,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-15 02:56:04,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:56:04,313 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-15 02:56:04,313 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-15 02:56:04,313 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-15 02:56:04,313 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-15 02:56:04,313 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-15 02:56:04,313 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-15 02:56:04,313 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-15 02:56:04,313 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-15 02:56:04,313 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681527356526.20e761db5975fb9082f870bf056bd078. 2023-04-15 02:56:04,313 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681527356526.20e761db5975fb9082f870bf056bd078. started... 2023-04-15 02:56:04,314 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681527356526.20e761db5975fb9082f870bf056bd078. as already flushing 2023-04-15 02:56:04,314 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681527356526.20e761db5975fb9082f870bf056bd078. 2023-04-15 02:56:04,314 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-15 02:56:04,314 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-15 02:56:04,314 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:04,314 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-15 02:56:04,314 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34681,1681527209133' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-15 02:56:04,314 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:56:04,314 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:04,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:04,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:56:04,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:56:04,315 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:56:04,315 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-15 02:56:04,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:56:04,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:56:04,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:56:04,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:04,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:56:04,316 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34681,1681527209133' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-15 02:56:04,316 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5f346aba[Count = 0] remaining members to acquire global barrier 2023-04-15 02:56:04,316 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-15 02:56:04,316 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:56:04,316 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:56:04,316 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:56:04,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:56:04,316 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:04,316 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-15 02:56:04,316 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-15 02:56:04,316 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-15 02:56:04,316 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34681,1681527209133' in zk 2023-04-15 02:56:04,317 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:04,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:04,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:56:04,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:56:04,317 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-15 02:56:04,317 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-15 02:56:04,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:56:04,317 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-15 02:56:04,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:56:04,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:56:04,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:04,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:56:04,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:56:04,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:04,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34681,1681527209133': 2023-04-15 02:56:04,319 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34681,1681527209133' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-15 02:56:04,319 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-15 02:56:04,319 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-15 02:56:04,319 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-15 02:56:04,319 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-15 02:56:04,319 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-15 02:56:04,320 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:56:04,320 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:56:04,320 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:56:04,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:56:04,320 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-15 02:56:04,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:56:04,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:56:04,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:56:04,320 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-15 02:56:04,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-15 02:56:04,320 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:04,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:56:04,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:56:04,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:56:04,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:56:04,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:56:04,321 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:04,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:04,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:56:04,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:56:04,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:04,322 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:04,322 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:56:04,322 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:56:04,322 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-15 02:56:04,322 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-15 02:56:04,322 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-15 02:56:04,322 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-15 02:56:04,322 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:04,322 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-15 02:56:04,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-15 02:56:04,323 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-15 02:56:04,322 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-15 02:56:04,323 DEBUG [Thread-3439] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-15 02:56:04,322 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:56:04,323 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-15 02:56:04,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-15 02:56:04,323 DEBUG [Thread-3439] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-15 02:56:04,323 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:56:04,323 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:56:04,391 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=20e761db5975fb9082f870bf056bd078, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:56:04,391 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34681] ipc.CallRunner(144): callId: 57 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:48666 deadline: 1681527424389, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=20e761db5975fb9082f870bf056bd078, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:04,391 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=20e761db5975fb9082f870bf056bd078, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:56:04,391 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34681] ipc.CallRunner(144): callId: 60 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:48624 deadline: 1681527424389, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=20e761db5975fb9082f870bf056bd078, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:04,391 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=20e761db5975fb9082f870bf056bd078, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:56:04,392 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34681] ipc.CallRunner(144): callId: 60 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:48664 deadline: 1681527424390, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=20e761db5975fb9082f870bf056bd078, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:04,394 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=20e761db5975fb9082f870bf056bd078, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:56:04,394 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34681] ipc.CallRunner(144): callId: 62 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:48586 deadline: 1681527424393, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=20e761db5975fb9082f870bf056bd078, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:04,396 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=20e761db5975fb9082f870bf056bd078, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:56:04,396 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34681] ipc.CallRunner(144): callId: 64 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:48648 deadline: 1681527424395, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=20e761db5975fb9082f870bf056bd078, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:04,423 DEBUG [Thread-3439] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-15 02:56:04,423 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-15 02:56:04,424 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-15 02:56:04,425 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-15 02:56:04,425 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-15 02:56:04,425 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-15 02:56:04,425 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-15 02:56:04,425 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-15 02:56:04,425 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-15 02:56:04,426 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:56:04,426 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:56:04,426 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:04,426 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-15 02:56:04,426 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-15 02:56:04,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-15 02:56:04,427 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:04,427 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-15 02:56:04,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:56:04,427 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:56:04,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-15 02:56:04,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:56:04,427 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-15 02:56:04,427 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-15 02:56:04,427 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-15 02:56:04,427 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-15 02:56:04,428 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-15 02:56:04,428 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-15 02:56:04,428 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-15 02:56:04,428 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-15 02:56:04,428 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681527356526.20e761db5975fb9082f870bf056bd078. 2023-04-15 02:56:04,428 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681527356526.20e761db5975fb9082f870bf056bd078. started... 2023-04-15 02:56:04,428 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681527356526.20e761db5975fb9082f870bf056bd078. as already flushing 2023-04-15 02:56:04,428 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681527356526.20e761db5975fb9082f870bf056bd078. 2023-04-15 02:56:04,428 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-15 02:56:04,428 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-15 02:56:04,428 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:04,428 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-15 02:56:04,428 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34681,1681527209133' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-15 02:56:04,429 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:04,429 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:56:04,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:04,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:56:04,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:56:04,429 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:56:04,429 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-15 02:56:04,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:56:04,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:56:04,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:56:04,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:04,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:56:04,430 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34681,1681527209133' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-15 02:56:04,430 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-15 02:56:04,430 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@b469101[Count = 0] remaining members to acquire global barrier 2023-04-15 02:56:04,430 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:56:04,431 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:56:04,431 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:56:04,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:56:04,431 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-15 02:56:04,431 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-15 02:56:04,431 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34681,1681527209133' in zk 2023-04-15 02:56:04,431 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:04,431 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-15 02:56:04,431 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-15 02:56:04,431 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-15 02:56:04,431 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:04,431 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-15 02:56:04,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:04,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:56:04,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:56:04,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:56:04,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:56:04,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:56:04,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:04,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:56:04,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:56:04,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:04,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34681,1681527209133': 2023-04-15 02:56:04,433 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34681,1681527209133' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-15 02:56:04,433 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-15 02:56:04,433 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-15 02:56:04,433 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-15 02:56:04,433 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-15 02:56:04,433 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-15 02:56:04,434 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:56:04,434 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:56:04,434 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:56:04,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:56:04,434 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-15 02:56:04,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:56:04,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:56:04,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:56:04,435 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-15 02:56:04,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-15 02:56:04,435 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:04,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:56:04,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:56:04,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:56:04,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:56:04,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:56:04,435 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:04,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:04,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:56:04,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:56:04,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:04,437 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:04,437 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:56:04,437 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-15 02:56:04,437 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:56:04,437 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-15 02:56:04,437 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-15 02:56:04,437 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-15 02:56:04,437 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-15 02:56:04,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-15 02:56:04,437 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-15 02:56:04,437 DEBUG [Thread-3439] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-15 02:56:04,438 DEBUG [Thread-3439] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-15 02:56:04,437 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:04,437 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-15 02:56:04,438 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:56:04,438 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-15 02:56:04,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-15 02:56:04,438 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:56:04,438 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:56:04,483 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=40.25 KB at sequenceid=120 (bloomFilter=true), to=hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/20e761db5975fb9082f870bf056bd078/.tmp/B/d3cc26ffda714b4786f1a1389190f5d5 2023-04-15 02:56:04,538 DEBUG [Thread-3439] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-15 02:56:04,538 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-15 02:56:04,539 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-15 02:56:04,540 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-15 02:56:04,540 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-15 02:56:04,540 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-15 02:56:04,540 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-15 02:56:04,540 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-15 02:56:04,540 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-15 02:56:04,541 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:56:04,541 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:56:04,541 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-15 02:56:04,541 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:04,541 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-15 02:56:04,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-15 02:56:04,542 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:04,542 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-15 02:56:04,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:56:04,542 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:56:04,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-15 02:56:04,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:56:04,542 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-15 02:56:04,542 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-15 02:56:04,542 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-15 02:56:04,542 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-15 02:56:04,543 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-15 02:56:04,543 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-15 02:56:04,543 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-15 02:56:04,543 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-15 02:56:04,543 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681527356526.20e761db5975fb9082f870bf056bd078. 2023-04-15 02:56:04,543 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681527356526.20e761db5975fb9082f870bf056bd078. started... 2023-04-15 02:56:04,543 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681527356526.20e761db5975fb9082f870bf056bd078. as already flushing 2023-04-15 02:56:04,543 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681527356526.20e761db5975fb9082f870bf056bd078. 2023-04-15 02:56:04,543 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-15 02:56:04,543 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-15 02:56:04,543 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:04,543 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-15 02:56:04,543 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34681,1681527209133' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-15 02:56:04,544 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:56:04,544 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:04,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:04,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:56:04,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:56:04,544 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:56:04,544 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-15 02:56:04,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:56:04,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:56:04,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:56:04,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:04,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:56:04,545 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34681,1681527209133' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-15 02:56:04,545 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-15 02:56:04,545 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1795d080[Count = 0] remaining members to acquire global barrier 2023-04-15 02:56:04,545 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:56:04,545 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:56:04,545 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:56:04,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:56:04,545 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-15 02:56:04,545 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-15 02:56:04,545 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:04,545 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-15 02:56:04,545 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34681,1681527209133' in zk 2023-04-15 02:56:04,546 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-15 02:56:04,546 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-15 02:56:04,546 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:04,546 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-15 02:56:04,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:04,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:56:04,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:56:04,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:56:04,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:56:04,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:56:04,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:04,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:56:04,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:56:04,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:04,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34681,1681527209133': 2023-04-15 02:56:04,548 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34681,1681527209133' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-15 02:56:04,548 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-15 02:56:04,548 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-15 02:56:04,548 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-15 02:56:04,548 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-15 02:56:04,548 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-15 02:56:04,549 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:56:04,549 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-15 02:56:04,549 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:56:04,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:56:04,549 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:56:04,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:56:04,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:56:04,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:56:04,550 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-15 02:56:04,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-15 02:56:04,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:56:04,550 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:04,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:56:04,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:56:04,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:56:04,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:56:04,551 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:04,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:04,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:56:04,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:56:04,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:04,552 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:04,552 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:56:04,552 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-15 02:56:04,552 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:56:04,552 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-15 02:56:04,552 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-15 02:56:04,552 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-15 02:56:04,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-15 02:56:04,552 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-15 02:56:04,553 DEBUG [Thread-3439] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-15 02:56:04,552 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-15 02:56:04,553 DEBUG [Thread-3439] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-15 02:56:04,552 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-15 02:56:04,552 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:04,553 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-15 02:56:04,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-15 02:56:04,553 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:56:04,553 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:56:04,553 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:56:04,653 DEBUG [Thread-3439] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-15 02:56:04,653 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-15 02:56:04,654 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-15 02:56:04,655 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-15 02:56:04,655 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-15 02:56:04,655 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-15 02:56:04,655 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-15 02:56:04,655 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-15 02:56:04,655 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-15 02:56:04,656 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:56:04,656 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:56:04,657 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-15 02:56:04,657 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:04,657 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-15 02:56:04,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-15 02:56:04,657 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:04,657 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-15 02:56:04,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:56:04,658 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:56:04,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-15 02:56:04,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:56:04,658 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-15 02:56:04,658 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-15 02:56:04,658 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-15 02:56:04,658 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-15 02:56:04,658 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-15 02:56:04,658 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-15 02:56:04,658 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-15 02:56:04,658 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-15 02:56:04,659 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681527356526.20e761db5975fb9082f870bf056bd078. 2023-04-15 02:56:04,659 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681527356526.20e761db5975fb9082f870bf056bd078. started... 2023-04-15 02:56:04,659 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681527356526.20e761db5975fb9082f870bf056bd078. as already flushing 2023-04-15 02:56:04,659 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681527356526.20e761db5975fb9082f870bf056bd078. 2023-04-15 02:56:04,659 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-15 02:56:04,659 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-15 02:56:04,659 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:04,659 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-15 02:56:04,659 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34681,1681527209133' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-15 02:56:04,659 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:04,659 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:56:04,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:04,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:56:04,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:56:04,660 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:56:04,660 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-15 02:56:04,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:56:04,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:56:04,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:56:04,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:04,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:56:04,661 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34681,1681527209133' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-15 02:56:04,661 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-15 02:56:04,661 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@611bb345[Count = 0] remaining members to acquire global barrier 2023-04-15 02:56:04,661 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:56:04,661 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:56:04,661 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:56:04,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:56:04,661 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-15 02:56:04,661 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-15 02:56:04,661 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:04,662 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-15 02:56:04,661 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34681,1681527209133' in zk 2023-04-15 02:56:04,662 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:04,662 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-15 02:56:04,662 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-15 02:56:04,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:04,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:56:04,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:56:04,662 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-15 02:56:04,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:56:04,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:56:04,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:56:04,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:04,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:56:04,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:56:04,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:04,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34681,1681527209133': 2023-04-15 02:56:04,664 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34681,1681527209133' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-15 02:56:04,664 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-15 02:56:04,664 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-15 02:56:04,664 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-15 02:56:04,665 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-15 02:56:04,665 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-15 02:56:04,665 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:56:04,665 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:56:04,665 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-15 02:56:04,665 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:56:04,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:56:04,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:56:04,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:56:04,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:56:04,666 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-15 02:56:04,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-15 02:56:04,666 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:04,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:56:04,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:56:04,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:56:04,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:56:04,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:56:04,667 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:04,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:04,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:56:04,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:56:04,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:04,668 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:04,668 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:56:04,668 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:56:04,668 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:04,668 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-15 02:56:04,668 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-15 02:56:04,668 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-15 02:56:04,668 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-15 02:56:04,668 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:56:04,668 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-15 02:56:04,668 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-15 02:56:04,668 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-15 02:56:04,668 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:56:04,668 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:56:04,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-15 02:56:04,668 DEBUG [Thread-3439] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-15 02:56:04,669 DEBUG [Thread-3439] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-15 02:56:04,669 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-15 02:56:04,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-15 02:56:04,769 DEBUG [Thread-3439] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-15 02:56:04,769 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-15 02:56:04,770 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-15 02:56:04,771 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-15 02:56:04,771 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-15 02:56:04,771 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-15 02:56:04,771 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-15 02:56:04,771 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-15 02:56:04,771 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-15 02:56:04,772 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:56:04,772 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:56:04,772 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:04,772 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-15 02:56:04,772 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-15 02:56:04,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-15 02:56:04,773 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:04,773 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-15 02:56:04,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:56:04,773 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:56:04,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-15 02:56:04,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:56:04,773 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-15 02:56:04,773 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-15 02:56:04,774 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-15 02:56:04,774 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-15 02:56:04,774 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-15 02:56:04,774 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-15 02:56:04,774 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-15 02:56:04,774 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-15 02:56:04,774 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681527356526.20e761db5975fb9082f870bf056bd078. 2023-04-15 02:56:04,774 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681527356526.20e761db5975fb9082f870bf056bd078. started... 2023-04-15 02:56:04,774 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681527356526.20e761db5975fb9082f870bf056bd078. as already flushing 2023-04-15 02:56:04,774 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681527356526.20e761db5975fb9082f870bf056bd078. 2023-04-15 02:56:04,774 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-15 02:56:04,774 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-15 02:56:04,774 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:04,774 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-15 02:56:04,774 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34681,1681527209133' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-15 02:56:04,775 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:56:04,775 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:04,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:04,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:56:04,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:56:04,775 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:56:04,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:56:04,775 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-15 02:56:04,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:56:04,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:56:04,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:04,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:56:04,776 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34681,1681527209133' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-15 02:56:04,776 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-15 02:56:04,776 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1d066466[Count = 0] remaining members to acquire global barrier 2023-04-15 02:56:04,776 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:56:04,777 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:56:04,777 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:56:04,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:56:04,777 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-15 02:56:04,777 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-15 02:56:04,777 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34681,1681527209133' in zk 2023-04-15 02:56:04,777 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:04,777 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-15 02:56:04,778 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:04,778 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-15 02:56:04,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:04,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:56:04,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:56:04,778 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-15 02:56:04,778 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-15 02:56:04,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:56:04,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:56:04,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:56:04,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:04,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:56:04,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:56:04,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:04,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34681,1681527209133': 2023-04-15 02:56:04,780 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34681,1681527209133' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-15 02:56:04,780 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-15 02:56:04,780 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-15 02:56:04,780 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-15 02:56:04,780 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-15 02:56:04,780 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-15 02:56:04,781 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:56:04,781 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:56:04,781 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-15 02:56:04,781 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:04,782 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:04,782 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:56:04,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:56:04,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:56:04,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:56:04,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:56:04,782 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-15 02:56:04,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-15 02:56:04,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:56:04,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:56:04,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:56:04,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:56:04,783 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-15 02:56:04,783 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-15 02:56:04,783 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-15 02:56:04,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-15 02:56:04,783 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-15 02:56:04,783 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:04,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:56:04,783 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-15 02:56:04,784 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-15 02:56:04,783 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:56:04,784 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-15 02:56:04,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-15 02:56:04,784 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:56:04,784 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:04,784 DEBUG [Thread-3439] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-15 02:56:04,784 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:56:04,784 DEBUG [Thread-3439] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-15 02:56:04,784 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-15 02:56:04,784 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:56:04,784 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:56:04,884 DEBUG [Thread-3439] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-15 02:56:04,884 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-15 02:56:04,886 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-15 02:56:04,887 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-15 02:56:04,887 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-15 02:56:04,887 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-15 02:56:04,887 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-15 02:56:04,887 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-15 02:56:04,887 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-15 02:56:04,887 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:56:04,887 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:56:04,888 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-15 02:56:04,888 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:04,888 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-15 02:56:04,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-15 02:56:04,888 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:04,888 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-15 02:56:04,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:56:04,889 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:56:04,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-15 02:56:04,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:56:04,889 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-15 02:56:04,889 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-15 02:56:04,889 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-15 02:56:04,889 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-15 02:56:04,889 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-15 02:56:04,889 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-15 02:56:04,890 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-15 02:56:04,890 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-15 02:56:04,890 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681527356526.20e761db5975fb9082f870bf056bd078. 2023-04-15 02:56:04,890 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681527356526.20e761db5975fb9082f870bf056bd078. started... 2023-04-15 02:56:04,890 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681527356526.20e761db5975fb9082f870bf056bd078. as already flushing 2023-04-15 02:56:04,890 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681527356526.20e761db5975fb9082f870bf056bd078. 2023-04-15 02:56:04,890 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-15 02:56:04,890 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-15 02:56:04,890 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:04,890 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-15 02:56:04,890 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34681,1681527209133' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-15 02:56:04,891 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:04,891 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:56:04,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:04,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:56:04,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:56:04,891 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:56:04,891 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-15 02:56:04,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:56:04,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:56:04,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:56:04,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:04,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:56:04,892 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=40.25 KB at sequenceid=120 (bloomFilter=true), to=hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/20e761db5975fb9082f870bf056bd078/.tmp/C/3d39f7c1089b41b48ed37ca52be51bb3 2023-04-15 02:56:04,894 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34681,1681527209133' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-15 02:56:04,894 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@22f7305e[Count = 0] remaining members to acquire global barrier 2023-04-15 02:56:04,894 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-15 02:56:04,894 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:56:04,894 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:56:04,894 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:56:04,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:56:04,895 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-15 02:56:04,895 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-15 02:56:04,895 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34681,1681527209133' in zk 2023-04-15 02:56:04,895 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:04,895 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-15 02:56:04,895 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-15 02:56:04,895 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:04,895 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-15 02:56:04,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:04,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:56:04,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:56:04,895 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-15 02:56:04,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:56:04,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:56:04,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:56:04,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:04,896 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/20e761db5975fb9082f870bf056bd078/.tmp/A/0312c634961f48589caed89d0a1783c0 as hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/20e761db5975fb9082f870bf056bd078/A/0312c634961f48589caed89d0a1783c0 2023-04-15 02:56:04,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:56:04,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:56:04,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:04,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34681,1681527209133': 2023-04-15 02:56:04,898 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34681,1681527209133' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-15 02:56:04,898 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-15 02:56:04,898 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-15 02:56:04,898 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-15 02:56:04,898 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-15 02:56:04,898 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-15 02:56:04,899 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:56:04,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:56:04,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:56:04,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:56:04,899 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:56:04,899 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-15 02:56:04,899 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/20e761db5975fb9082f870bf056bd078/A/0312c634961f48589caed89d0a1783c0, entries=150, sequenceid=120, filesize=25.1 K 2023-04-15 02:56:04,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:56:04,899 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:04,899 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:56:04,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:56:04,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:56:04,900 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-15 02:56:04,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-15 02:56:04,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:56:04,900 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/20e761db5975fb9082f870bf056bd078/.tmp/B/d3cc26ffda714b4786f1a1389190f5d5 as hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/20e761db5975fb9082f870bf056bd078/B/d3cc26ffda714b4786f1a1389190f5d5 2023-04-15 02:56:04,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:56:04,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:56:04,900 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:04,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:04,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:56:04,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:56:04,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:04,902 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:04,902 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:56:04,902 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:56:04,902 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:04,902 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:56:04,902 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-15 02:56:04,902 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-15 02:56:04,902 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-15 02:56:04,903 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-15 02:56:04,902 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:56:04,903 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:56:04,903 DEBUG [Thread-3439] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-15 02:56:04,903 DEBUG [Thread-3439] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-15 02:56:04,904 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-15 02:56:04,904 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-15 02:56:04,904 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-15 02:56:04,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-15 02:56:04,904 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-15 02:56:04,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-15 02:56:04,905 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/20e761db5975fb9082f870bf056bd078/B/d3cc26ffda714b4786f1a1389190f5d5, entries=150, sequenceid=120, filesize=11.6 K 2023-04-15 02:56:04,906 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/20e761db5975fb9082f870bf056bd078/.tmp/C/3d39f7c1089b41b48ed37ca52be51bb3 as hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/20e761db5975fb9082f870bf056bd078/C/3d39f7c1089b41b48ed37ca52be51bb3 2023-04-15 02:56:04,908 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/20e761db5975fb9082f870bf056bd078/C/3d39f7c1089b41b48ed37ca52be51bb3, entries=150, sequenceid=120, filesize=11.6 K 2023-04-15 02:56:04,909 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~120.76 KB/123660, heapSize ~360 KB/368640, currentSize=53.67 KB/54960 for 20e761db5975fb9082f870bf056bd078 in 1659ms, sequenceid=120, compaction requested=true 2023-04-15 02:56:04,909 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 20e761db5975fb9082f870bf056bd078: 2023-04-15 02:56:04,909 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-15 02:56:04,909 DEBUG [RS:0;jenkins-hbase20:34681-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-15 02:56:04,909 DEBUG [RS:0;jenkins-hbase20:34681-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-15 02:56:04,909 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-15 02:56:04,909 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-15 02:56:04,910 DEBUG [RS:0;jenkins-hbase20:34681-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 103087 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-15 02:56:04,910 DEBUG [RS:0;jenkins-hbase20:34681-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 47647 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-15 02:56:04,910 DEBUG [RS:0;jenkins-hbase20:34681-longCompactions-0] regionserver.HStore(1912): 20e761db5975fb9082f870bf056bd078/B is initiating minor compaction (all files) 2023-04-15 02:56:04,910 INFO [RS:0;jenkins-hbase20:34681-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 20e761db5975fb9082f870bf056bd078/B in TestAcidGuarantees,,1681527356526.20e761db5975fb9082f870bf056bd078. 2023-04-15 02:56:04,910 INFO [RS:0;jenkins-hbase20:34681-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/20e761db5975fb9082f870bf056bd078/B/0969d679f2bb4597bac256d78f824067, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/20e761db5975fb9082f870bf056bd078/B/0c295b6fa6c34914ba5164667243e923, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/20e761db5975fb9082f870bf056bd078/B/c030a2631f8f4bc0b69a4c216dc58a93, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/20e761db5975fb9082f870bf056bd078/B/d3cc26ffda714b4786f1a1389190f5d5] into tmpdir=hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/20e761db5975fb9082f870bf056bd078/.tmp, totalSize=46.5 K 2023-04-15 02:56:04,910 DEBUG [RS:0;jenkins-hbase20:34681-shortCompactions-0] regionserver.HStore(1912): 20e761db5975fb9082f870bf056bd078/A is initiating minor compaction (all files) 2023-04-15 02:56:04,910 INFO [RS:0;jenkins-hbase20:34681-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 20e761db5975fb9082f870bf056bd078/A in TestAcidGuarantees,,1681527356526.20e761db5975fb9082f870bf056bd078. 2023-04-15 02:56:04,910 INFO [RS:0;jenkins-hbase20:34681-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/20e761db5975fb9082f870bf056bd078/A/7f2af62291f44d66a5b3f7a3ff162745, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/20e761db5975fb9082f870bf056bd078/A/682a33ab364b4412a3f05a564e0c8208, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/20e761db5975fb9082f870bf056bd078/A/61964aa8922847d490712f3d4966e25c, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/20e761db5975fb9082f870bf056bd078/A/0312c634961f48589caed89d0a1783c0] into tmpdir=hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/20e761db5975fb9082f870bf056bd078/.tmp, totalSize=100.7 K 2023-04-15 02:56:04,910 DEBUG [RS:0;jenkins-hbase20:34681-longCompactions-0] compactions.Compactor(207): Compacting 0969d679f2bb4597bac256d78f824067, keycount=150, bloomtype=ROW, size=11.7 K, encoding=NONE, compression=NONE, seqNum=64, earliestPutTs=1681527359363 2023-04-15 02:56:04,911 DEBUG [RS:0;jenkins-hbase20:34681-shortCompactions-0] compactions.Compactor(207): Compacting 7f2af62291f44d66a5b3f7a3ff162745, keycount=150, bloomtype=ROW, size=25.2 K, encoding=NONE, compression=NONE, seqNum=64, earliestPutTs=1681527359363 2023-04-15 02:56:04,911 DEBUG [RS:0;jenkins-hbase20:34681-longCompactions-0] compactions.Compactor(207): Compacting 0c295b6fa6c34914ba5164667243e923, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=83, earliestPutTs=1681527360489 2023-04-15 02:56:04,911 DEBUG [RS:0;jenkins-hbase20:34681-shortCompactions-0] compactions.Compactor(207): Compacting 682a33ab364b4412a3f05a564e0c8208, keycount=150, bloomtype=ROW, size=25.1 K, encoding=NONE, compression=NONE, seqNum=83, earliestPutTs=1681527360489 2023-04-15 02:56:04,911 DEBUG [RS:0;jenkins-hbase20:34681-longCompactions-0] compactions.Compactor(207): Compacting c030a2631f8f4bc0b69a4c216dc58a93, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=99, earliestPutTs=1681527361105 2023-04-15 02:56:04,911 DEBUG [RS:0;jenkins-hbase20:34681-shortCompactions-0] compactions.Compactor(207): Compacting 61964aa8922847d490712f3d4966e25c, keycount=150, bloomtype=ROW, size=25.1 K, encoding=NONE, compression=NONE, seqNum=99, earliestPutTs=1681527361105 2023-04-15 02:56:04,911 DEBUG [RS:0;jenkins-hbase20:34681-longCompactions-0] compactions.Compactor(207): Compacting d3cc26ffda714b4786f1a1389190f5d5, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=120, earliestPutTs=1681527363236 2023-04-15 02:56:04,911 DEBUG [RS:0;jenkins-hbase20:34681-shortCompactions-0] compactions.Compactor(207): Compacting 0312c634961f48589caed89d0a1783c0, keycount=150, bloomtype=ROW, size=25.1 K, encoding=NONE, compression=NONE, seqNum=120, earliestPutTs=1681527363236 2023-04-15 02:56:04,917 INFO [RS:0;jenkins-hbase20:34681-longCompactions-0] throttle.PressureAwareThroughputController(145): 20e761db5975fb9082f870bf056bd078#B#compaction#462 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-15 02:56:04,920 INFO [RS:0;jenkins-hbase20:34681-shortCompactions-0] throttle.PressureAwareThroughputController(145): 20e761db5975fb9082f870bf056bd078#A#compaction#461 average throughput is 4.93 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-15 02:56:04,928 DEBUG [RS:0;jenkins-hbase20:34681-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/20e761db5975fb9082f870bf056bd078/.tmp/B/f092b0e0f6b44cb280f40a09566e7c43 as hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/20e761db5975fb9082f870bf056bd078/B/f092b0e0f6b44cb280f40a09566e7c43 2023-04-15 02:56:04,931 INFO [RS:0;jenkins-hbase20:34681-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 20e761db5975fb9082f870bf056bd078/B of 20e761db5975fb9082f870bf056bd078 into f092b0e0f6b44cb280f40a09566e7c43(size=11.8 K), total size for store is 11.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-15 02:56:04,931 DEBUG [RS:0;jenkins-hbase20:34681-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 20e761db5975fb9082f870bf056bd078: 2023-04-15 02:56:04,931 INFO [RS:0;jenkins-hbase20:34681-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681527356526.20e761db5975fb9082f870bf056bd078., storeName=20e761db5975fb9082f870bf056bd078/B, priority=12, startTime=1681527364909; duration=0sec 2023-04-15 02:56:04,931 DEBUG [RS:0;jenkins-hbase20:34681-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-15 02:56:04,931 DEBUG [RS:0;jenkins-hbase20:34681-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-15 02:56:04,932 DEBUG [RS:0;jenkins-hbase20:34681-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 47647 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-15 02:56:04,932 DEBUG [RS:0;jenkins-hbase20:34681-longCompactions-0] regionserver.HStore(1912): 20e761db5975fb9082f870bf056bd078/C is initiating minor compaction (all files) 2023-04-15 02:56:04,932 INFO [RS:0;jenkins-hbase20:34681-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 20e761db5975fb9082f870bf056bd078/C in TestAcidGuarantees,,1681527356526.20e761db5975fb9082f870bf056bd078. 2023-04-15 02:56:04,932 INFO [RS:0;jenkins-hbase20:34681-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/20e761db5975fb9082f870bf056bd078/C/6dd346ee5c4e4a1f8bf4d4dff5b0287c, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/20e761db5975fb9082f870bf056bd078/C/06699938be04491a822b2e05301c55a8, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/20e761db5975fb9082f870bf056bd078/C/6c289f99aeaa4124bd3fd318af17610c, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/20e761db5975fb9082f870bf056bd078/C/3d39f7c1089b41b48ed37ca52be51bb3] into tmpdir=hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/20e761db5975fb9082f870bf056bd078/.tmp, totalSize=46.5 K 2023-04-15 02:56:04,932 DEBUG [RS:0;jenkins-hbase20:34681-longCompactions-0] compactions.Compactor(207): Compacting 6dd346ee5c4e4a1f8bf4d4dff5b0287c, keycount=150, bloomtype=ROW, size=11.7 K, encoding=NONE, compression=NONE, seqNum=64, earliestPutTs=1681527359363 2023-04-15 02:56:04,932 DEBUG [RS:0;jenkins-hbase20:34681-longCompactions-0] compactions.Compactor(207): Compacting 06699938be04491a822b2e05301c55a8, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=83, earliestPutTs=1681527360489 2023-04-15 02:56:04,933 DEBUG [RS:0;jenkins-hbase20:34681-longCompactions-0] compactions.Compactor(207): Compacting 6c289f99aeaa4124bd3fd318af17610c, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=99, earliestPutTs=1681527361105 2023-04-15 02:56:04,933 DEBUG [RS:0;jenkins-hbase20:34681-longCompactions-0] compactions.Compactor(207): Compacting 3d39f7c1089b41b48ed37ca52be51bb3, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=120, earliestPutTs=1681527363236 2023-04-15 02:56:04,937 INFO [RS:0;jenkins-hbase20:34681-longCompactions-0] throttle.PressureAwareThroughputController(145): 20e761db5975fb9082f870bf056bd078#C#compaction#463 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-15 02:56:05,003 DEBUG [Thread-3439] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-15 02:56:05,004 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-15 02:56:05,005 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-15 02:56:05,006 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-15 02:56:05,006 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-15 02:56:05,006 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-15 02:56:05,006 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-15 02:56:05,006 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-15 02:56:05,006 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-15 02:56:05,006 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:56:05,006 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:56:05,007 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-15 02:56:05,007 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:05,007 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-15 02:56:05,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-15 02:56:05,007 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:05,007 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-15 02:56:05,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:56:05,008 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:56:05,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-15 02:56:05,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:56:05,009 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-15 02:56:05,009 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-15 02:56:05,009 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-15 02:56:05,009 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-15 02:56:05,009 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-15 02:56:05,009 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-15 02:56:05,009 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-15 02:56:05,009 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-15 02:56:05,009 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681527356526.20e761db5975fb9082f870bf056bd078. 2023-04-15 02:56:05,009 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681527356526.20e761db5975fb9082f870bf056bd078. started... 2023-04-15 02:56:05,010 INFO [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 20e761db5975fb9082f870bf056bd078 3/3 column families, dataSize=53.67 KB heapSize=160.13 KB 2023-04-15 02:56:05,010 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 20e761db5975fb9082f870bf056bd078, store=A 2023-04-15 02:56:05,010 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-15 02:56:05,010 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 20e761db5975fb9082f870bf056bd078, store=B 2023-04-15 02:56:05,010 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-15 02:56:05,010 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 20e761db5975fb9082f870bf056bd078, store=C 2023-04-15 02:56:05,010 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-15 02:56:05,346 DEBUG [RS:0;jenkins-hbase20:34681-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/20e761db5975fb9082f870bf056bd078/.tmp/C/138a4607009640ba9d4821f3f7e6e712 as hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/20e761db5975fb9082f870bf056bd078/C/138a4607009640ba9d4821f3f7e6e712 2023-04-15 02:56:05,349 INFO [RS:0;jenkins-hbase20:34681-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 20e761db5975fb9082f870bf056bd078/C of 20e761db5975fb9082f870bf056bd078 into 138a4607009640ba9d4821f3f7e6e712(size=11.8 K), total size for store is 11.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-15 02:56:05,349 DEBUG [RS:0;jenkins-hbase20:34681-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 20e761db5975fb9082f870bf056bd078: 2023-04-15 02:56:05,349 INFO [RS:0;jenkins-hbase20:34681-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681527356526.20e761db5975fb9082f870bf056bd078., storeName=20e761db5975fb9082f870bf056bd078/C, priority=12, startTime=1681527364909; duration=0sec 2023-04-15 02:56:05,349 DEBUG [RS:0;jenkins-hbase20:34681-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-15 02:56:05,392 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34681] regionserver.HRegion(9158): Flush requested on 20e761db5975fb9082f870bf056bd078 2023-04-15 02:56:05,393 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681527356526.20e761db5975fb9082f870bf056bd078. as already flushing 2023-04-15 02:56:05,419 INFO [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230415d8f2a6ce669d4ed88252e79e7cb275a4 to hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230415d8f2a6ce669d4ed88252e79e7cb275a4 2023-04-15 02:56:05,429 INFO [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=132, memsize=17.9 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/20e761db5975fb9082f870bf056bd078/.tmp/A/835c5281097746cb9ba12cc89ec8adbe 2023-04-15 02:56:05,434 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=20e761db5975fb9082f870bf056bd078, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:56:05,434 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34681] ipc.CallRunner(144): callId: 68 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:48648 deadline: 1681527425429, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=20e761db5975fb9082f870bf056bd078, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:05,434 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=20e761db5975fb9082f870bf056bd078, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:56:05,434 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34681] ipc.CallRunner(144): callId: 65 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:48664 deadline: 1681527425429, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=20e761db5975fb9082f870bf056bd078, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:05,434 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=20e761db5975fb9082f870bf056bd078, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:56:05,434 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34681] ipc.CallRunner(144): callId: 67 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:48624 deadline: 1681527425429, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=20e761db5975fb9082f870bf056bd078, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:05,434 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=20e761db5975fb9082f870bf056bd078, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:56:05,435 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34681] ipc.CallRunner(144): callId: 65 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:48666 deadline: 1681527425431, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=20e761db5975fb9082f870bf056bd078, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:05,435 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681527365403, max timestamp=1681527365433Num uniques 859; 2023-04-15 02:56:05,435 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-15 02:56:05,435 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681527365403, max timestamp=1681527365433Num uniques 859; 2023-04-15 02:56:05,435 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681527365403, max timestamp=1681527365433Num uniques 859; 2023-04-15 02:56:05,435 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-3] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-15 02:56:05,435 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-9] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-15 02:56:05,487 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=20e761db5975fb9082f870bf056bd078, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:56:05,487 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34681] ipc.CallRunner(144): callId: 81 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:48586 deadline: 1681527425484, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=20e761db5975fb9082f870bf056bd078, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:05,538 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=20e761db5975fb9082f870bf056bd078, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:56:05,538 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34681] ipc.CallRunner(144): callId: 67 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:48666 deadline: 1681527425536, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=20e761db5975fb9082f870bf056bd078, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:05,538 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=20e761db5975fb9082f870bf056bd078, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:56:05,538 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34681] ipc.CallRunner(144): callId: 67 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:48664 deadline: 1681527425536, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=20e761db5975fb9082f870bf056bd078, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:05,538 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=20e761db5975fb9082f870bf056bd078, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:56:05,538 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34681] ipc.CallRunner(144): callId: 70 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:48648 deadline: 1681527425536, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=20e761db5975fb9082f870bf056bd078, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:05,539 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=20e761db5975fb9082f870bf056bd078, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:56:05,539 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34681] ipc.CallRunner(144): callId: 69 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:48624 deadline: 1681527425536, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=20e761db5975fb9082f870bf056bd078, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:05,589 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=20e761db5975fb9082f870bf056bd078, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:56:05,590 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34681] ipc.CallRunner(144): callId: 83 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:48586 deadline: 1681527425588, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=20e761db5975fb9082f870bf056bd078, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:05,740 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=20e761db5975fb9082f870bf056bd078, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:56:05,740 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34681] ipc.CallRunner(144): callId: 69 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:48664 deadline: 1681527425739, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=20e761db5975fb9082f870bf056bd078, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:05,741 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=20e761db5975fb9082f870bf056bd078, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:56:05,741 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34681] ipc.CallRunner(144): callId: 71 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:48624 deadline: 1681527425739, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=20e761db5975fb9082f870bf056bd078, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:05,741 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=20e761db5975fb9082f870bf056bd078, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:56:05,742 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34681] ipc.CallRunner(144): callId: 69 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:48666 deadline: 1681527425740, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=20e761db5975fb9082f870bf056bd078, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:05,742 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=20e761db5975fb9082f870bf056bd078, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:56:05,742 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34681] ipc.CallRunner(144): callId: 72 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:48648 deadline: 1681527425740, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=20e761db5975fb9082f870bf056bd078, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:05,792 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=20e761db5975fb9082f870bf056bd078, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:56:05,792 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34681] ipc.CallRunner(144): callId: 85 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:48586 deadline: 1681527425790, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=20e761db5975fb9082f870bf056bd078, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:05,838 INFO [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=132 (bloomFilter=true), to=hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/20e761db5975fb9082f870bf056bd078/.tmp/B/13265909e92d49d793b4cf103df530a3 2023-04-15 02:56:06,041 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=20e761db5975fb9082f870bf056bd078, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:56:06,041 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34681] ipc.CallRunner(144): callId: 71 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:48664 deadline: 1681527426041, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=20e761db5975fb9082f870bf056bd078, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:06,044 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=20e761db5975fb9082f870bf056bd078, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:56:06,044 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34681] ipc.CallRunner(144): callId: 73 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:48624 deadline: 1681527426043, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=20e761db5975fb9082f870bf056bd078, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:06,045 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=20e761db5975fb9082f870bf056bd078, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:56:06,045 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34681] ipc.CallRunner(144): callId: 71 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:48666 deadline: 1681527426043, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=20e761db5975fb9082f870bf056bd078, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:06,046 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=20e761db5975fb9082f870bf056bd078, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:56:06,047 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34681] ipc.CallRunner(144): callId: 74 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:48648 deadline: 1681527426044, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=20e761db5975fb9082f870bf056bd078, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:06,095 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=20e761db5975fb9082f870bf056bd078, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:56:06,095 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34681] ipc.CallRunner(144): callId: 87 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:48586 deadline: 1681527426094, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=20e761db5975fb9082f870bf056bd078, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:06,138 DEBUG [RS:0;jenkins-hbase20:34681-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/20e761db5975fb9082f870bf056bd078/.tmp/A/1839c025181b488a9f7e760e76b34ff6 as hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/20e761db5975fb9082f870bf056bd078/A/1839c025181b488a9f7e760e76b34ff6 2023-04-15 02:56:06,142 INFO [RS:0;jenkins-hbase20:34681-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 20e761db5975fb9082f870bf056bd078/A of 20e761db5975fb9082f870bf056bd078 into 1839c025181b488a9f7e760e76b34ff6(size=25.4 K), total size for store is 25.4 K. This selection was in queue for 0sec, and took 1sec to execute. 2023-04-15 02:56:06,143 DEBUG [RS:0;jenkins-hbase20:34681-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 20e761db5975fb9082f870bf056bd078: 2023-04-15 02:56:06,143 INFO [RS:0;jenkins-hbase20:34681-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681527356526.20e761db5975fb9082f870bf056bd078., storeName=20e761db5975fb9082f870bf056bd078/A, priority=12, startTime=1681527364909; duration=1sec 2023-04-15 02:56:06,143 DEBUG [RS:0;jenkins-hbase20:34681-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-15 02:56:06,246 INFO [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=132 (bloomFilter=true), to=hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/20e761db5975fb9082f870bf056bd078/.tmp/C/def7bce379f0445f9fdfd2b2f011416a 2023-04-15 02:56:06,250 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/20e761db5975fb9082f870bf056bd078/.tmp/A/835c5281097746cb9ba12cc89ec8adbe as hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/20e761db5975fb9082f870bf056bd078/A/835c5281097746cb9ba12cc89ec8adbe 2023-04-15 02:56:06,253 INFO [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/20e761db5975fb9082f870bf056bd078/A/835c5281097746cb9ba12cc89ec8adbe, entries=150, sequenceid=132, filesize=25.2 K 2023-04-15 02:56:06,253 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/20e761db5975fb9082f870bf056bd078/.tmp/B/13265909e92d49d793b4cf103df530a3 as hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/20e761db5975fb9082f870bf056bd078/B/13265909e92d49d793b4cf103df530a3 2023-04-15 02:56:06,261 INFO [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/20e761db5975fb9082f870bf056bd078/B/13265909e92d49d793b4cf103df530a3, entries=150, sequenceid=132, filesize=11.7 K 2023-04-15 02:56:06,261 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/20e761db5975fb9082f870bf056bd078/.tmp/C/def7bce379f0445f9fdfd2b2f011416a as hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/20e761db5975fb9082f870bf056bd078/C/def7bce379f0445f9fdfd2b2f011416a 2023-04-15 02:56:06,264 INFO [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/20e761db5975fb9082f870bf056bd078/C/def7bce379f0445f9fdfd2b2f011416a, entries=150, sequenceid=132, filesize=11.7 K 2023-04-15 02:56:06,265 INFO [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=127.47 KB/130530 for 20e761db5975fb9082f870bf056bd078 in 1255ms, sequenceid=132, compaction requested=false 2023-04-15 02:56:06,265 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 20e761db5975fb9082f870bf056bd078: 2023-04-15 02:56:06,265 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681527356526.20e761db5975fb9082f870bf056bd078. 2023-04-15 02:56:06,265 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-15 02:56:06,265 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-15 02:56:06,265 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:06,265 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-15 02:56:06,265 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34681,1681527209133' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-15 02:56:06,267 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:06,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:06,267 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:56:06,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:56:06,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:56:06,267 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:56:06,267 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-15 02:56:06,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:56:06,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:56:06,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:56:06,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:06,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:56:06,268 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34681,1681527209133' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-15 02:56:06,268 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1b5ff550[Count = 0] remaining members to acquire global barrier 2023-04-15 02:56:06,268 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-15 02:56:06,268 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:56:06,269 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:56:06,269 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:56:06,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:56:06,269 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-15 02:56:06,269 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-15 02:56:06,269 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34681,1681527209133' in zk 2023-04-15 02:56:06,270 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:06,270 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-15 02:56:06,270 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:06,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:06,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:56:06,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:56:06,270 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-15 02:56:06,270 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-15 02:56:06,270 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-15 02:56:06,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:56:06,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:56:06,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:56:06,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:06,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:56:06,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:56:06,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:06,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34681,1681527209133': 2023-04-15 02:56:06,272 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34681,1681527209133' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-15 02:56:06,272 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-15 02:56:06,272 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-15 02:56:06,272 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-15 02:56:06,272 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-15 02:56:06,272 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-15 02:56:06,281 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:56:06,281 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:56:06,281 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-15 02:56:06,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:56:06,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:56:06,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:56:06,281 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:56:06,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:56:06,281 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:06,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:56:06,282 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-15 02:56:06,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-15 02:56:06,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:56:06,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:56:06,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:56:06,283 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:06,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:56:06,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:06,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:56:06,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:56:06,284 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-15 02:56:06,284 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:06,284 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-15 02:56:06,284 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-15 02:56:06,284 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-15 02:56:06,284 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-15 02:56:06,284 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-15 02:56:06,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-15 02:56:06,284 DEBUG [Thread-3439] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-15 02:56:06,284 DEBUG [Thread-3439] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-15 02:56:06,284 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:56:06,284 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-15 02:56:06,284 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:56:06,284 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-15 02:56:06,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-15 02:56:06,284 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:06,285 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:56:06,285 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:56:06,285 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:56:06,384 DEBUG [Thread-3439] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-15 02:56:06,385 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-15 02:56:06,386 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-15 02:56:06,387 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-15 02:56:06,387 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-15 02:56:06,387 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-15 02:56:06,387 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-15 02:56:06,387 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-15 02:56:06,387 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-15 02:56:06,387 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:56:06,388 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:56:06,388 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-15 02:56:06,388 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-15 02:56:06,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-15 02:56:06,388 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:06,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:56:06,388 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:06,389 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-15 02:56:06,389 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:56:06,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-15 02:56:06,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:56:06,389 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-15 02:56:06,389 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-15 02:56:06,389 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-15 02:56:06,389 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-15 02:56:06,389 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-15 02:56:06,389 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-15 02:56:06,389 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-15 02:56:06,389 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-15 02:56:06,389 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681527356526.20e761db5975fb9082f870bf056bd078. 2023-04-15 02:56:06,390 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681527356526.20e761db5975fb9082f870bf056bd078. started... 2023-04-15 02:56:06,390 INFO [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 20e761db5975fb9082f870bf056bd078 3/3 column families, dataSize=127.47 KB heapSize=365.06 KB 2023-04-15 02:56:06,390 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 20e761db5975fb9082f870bf056bd078, store=A 2023-04-15 02:56:06,390 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-15 02:56:06,390 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 20e761db5975fb9082f870bf056bd078, store=B 2023-04-15 02:56:06,390 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-15 02:56:06,390 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 20e761db5975fb9082f870bf056bd078, store=C 2023-04-15 02:56:06,390 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-15 02:56:06,545 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34681] regionserver.HRegion(9158): Flush requested on 20e761db5975fb9082f870bf056bd078 2023-04-15 02:56:06,545 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681527356526.20e761db5975fb9082f870bf056bd078. as already flushing 2023-04-15 02:56:06,554 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=20e761db5975fb9082f870bf056bd078, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:56:06,555 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34681] ipc.CallRunner(144): callId: 77 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:48648 deadline: 1681527426551, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=20e761db5975fb9082f870bf056bd078, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:06,555 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=20e761db5975fb9082f870bf056bd078, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:56:06,555 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34681] ipc.CallRunner(144): callId: 76 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:48664 deadline: 1681527426552, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=20e761db5975fb9082f870bf056bd078, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:06,558 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=20e761db5975fb9082f870bf056bd078, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:56:06,558 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34681] ipc.CallRunner(144): callId: 77 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:48624 deadline: 1681527426554, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=20e761db5975fb9082f870bf056bd078, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:06,558 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=20e761db5975fb9082f870bf056bd078, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:56:06,558 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34681] ipc.CallRunner(144): callId: 75 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:48666 deadline: 1681527426555, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=20e761db5975fb9082f870bf056bd078, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:06,601 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=20e761db5975fb9082f870bf056bd078, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:56:06,601 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34681] ipc.CallRunner(144): callId: 89 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:48586 deadline: 1681527426600, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=20e761db5975fb9082f870bf056bd078, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:06,657 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=20e761db5975fb9082f870bf056bd078, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:56:06,657 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34681] ipc.CallRunner(144): callId: 79 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:48648 deadline: 1681527426655, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=20e761db5975fb9082f870bf056bd078, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:06,657 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=20e761db5975fb9082f870bf056bd078, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:56:06,657 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34681] ipc.CallRunner(144): callId: 78 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:48664 deadline: 1681527426655, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=20e761db5975fb9082f870bf056bd078, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:06,660 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=20e761db5975fb9082f870bf056bd078, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:56:06,660 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34681] ipc.CallRunner(144): callId: 79 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:48624 deadline: 1681527426659, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=20e761db5975fb9082f870bf056bd078, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:06,660 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=20e761db5975fb9082f870bf056bd078, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:56:06,660 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34681] ipc.CallRunner(144): callId: 77 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:48666 deadline: 1681527426659, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=20e761db5975fb9082f870bf056bd078, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:06,800 INFO [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230415a7ba445cc5ae4e4cbfb7f9025e44832d to hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230415a7ba445cc5ae4e4cbfb7f9025e44832d 2023-04-15 02:56:06,859 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=20e761db5975fb9082f870bf056bd078, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:56:06,859 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34681] ipc.CallRunner(144): callId: 81 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:48648 deadline: 1681527426857, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=20e761db5975fb9082f870bf056bd078, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:06,860 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=20e761db5975fb9082f870bf056bd078, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:56:06,860 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34681] ipc.CallRunner(144): callId: 80 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:48664 deadline: 1681527426859, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=20e761db5975fb9082f870bf056bd078, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:06,863 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=20e761db5975fb9082f870bf056bd078, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:56:06,863 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34681] ipc.CallRunner(144): callId: 81 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:48624 deadline: 1681527426862, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=20e761db5975fb9082f870bf056bd078, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:06,864 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=20e761db5975fb9082f870bf056bd078, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:56:06,864 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34681] ipc.CallRunner(144): callId: 79 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:48666 deadline: 1681527426862, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=20e761db5975fb9082f870bf056bd078, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:07,162 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=20e761db5975fb9082f870bf056bd078, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:56:07,162 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34681] ipc.CallRunner(144): callId: 82 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:48664 deadline: 1681527427160, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=20e761db5975fb9082f870bf056bd078, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:07,164 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=20e761db5975fb9082f870bf056bd078, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:56:07,164 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34681] ipc.CallRunner(144): callId: 83 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:48648 deadline: 1681527427162, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=20e761db5975fb9082f870bf056bd078, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:07,166 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=20e761db5975fb9082f870bf056bd078, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:56:07,166 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34681] ipc.CallRunner(144): callId: 81 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:48666 deadline: 1681527427165, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=20e761db5975fb9082f870bf056bd078, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:07,167 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=20e761db5975fb9082f870bf056bd078, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:56:07,167 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34681] ipc.CallRunner(144): callId: 83 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:48624 deadline: 1681527427166, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=20e761db5975fb9082f870bf056bd078, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:07,205 INFO [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=170, memsize=42.5 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/20e761db5975fb9082f870bf056bd078/.tmp/A/168e7843b8c9470f857fedbb03a0eb8c 2023-04-15 02:56:07,611 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=20e761db5975fb9082f870bf056bd078, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:56:07,611 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34681] ipc.CallRunner(144): callId: 91 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:48586 deadline: 1681527427610, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=20e761db5975fb9082f870bf056bd078, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:07,614 INFO [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=170 (bloomFilter=true), to=hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/20e761db5975fb9082f870bf056bd078/.tmp/B/06e500fe23174a8896b3d0f9dd3a4863 2023-04-15 02:56:07,668 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=20e761db5975fb9082f870bf056bd078, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:56:07,668 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34681] ipc.CallRunner(144): callId: 84 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:48664 deadline: 1681527427667, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=20e761db5975fb9082f870bf056bd078, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:07,669 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=20e761db5975fb9082f870bf056bd078, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:56:07,669 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34681] ipc.CallRunner(144): callId: 85 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:48648 deadline: 1681527427667, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=20e761db5975fb9082f870bf056bd078, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:07,670 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=20e761db5975fb9082f870bf056bd078, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:56:07,671 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34681] ipc.CallRunner(144): callId: 83 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:48666 deadline: 1681527427669, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=20e761db5975fb9082f870bf056bd078, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:07,673 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=20e761db5975fb9082f870bf056bd078, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:56:07,673 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34681] ipc.CallRunner(144): callId: 85 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:48624 deadline: 1681527427671, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=20e761db5975fb9082f870bf056bd078, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:08,021 INFO [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=170 (bloomFilter=true), to=hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/20e761db5975fb9082f870bf056bd078/.tmp/C/633488e8312e4475b90ebf1ca8f775dd 2023-04-15 02:56:08,025 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/20e761db5975fb9082f870bf056bd078/.tmp/A/168e7843b8c9470f857fedbb03a0eb8c as hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/20e761db5975fb9082f870bf056bd078/A/168e7843b8c9470f857fedbb03a0eb8c 2023-04-15 02:56:08,027 INFO [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/20e761db5975fb9082f870bf056bd078/A/168e7843b8c9470f857fedbb03a0eb8c, entries=150, sequenceid=170, filesize=25.3 K 2023-04-15 02:56:08,028 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/20e761db5975fb9082f870bf056bd078/.tmp/B/06e500fe23174a8896b3d0f9dd3a4863 as hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/20e761db5975fb9082f870bf056bd078/B/06e500fe23174a8896b3d0f9dd3a4863 2023-04-15 02:56:08,031 INFO [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/20e761db5975fb9082f870bf056bd078/B/06e500fe23174a8896b3d0f9dd3a4863, entries=150, sequenceid=170, filesize=11.8 K 2023-04-15 02:56:08,031 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/20e761db5975fb9082f870bf056bd078/.tmp/C/633488e8312e4475b90ebf1ca8f775dd as hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/20e761db5975fb9082f870bf056bd078/C/633488e8312e4475b90ebf1ca8f775dd 2023-04-15 02:56:08,033 INFO [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/20e761db5975fb9082f870bf056bd078/C/633488e8312e4475b90ebf1ca8f775dd, entries=150, sequenceid=170, filesize=11.8 K 2023-04-15 02:56:08,034 INFO [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~127.47 KB/130530, heapSize ~365.02 KB/373776, currentSize=53.67 KB/54960 for 20e761db5975fb9082f870bf056bd078 in 1644ms, sequenceid=170, compaction requested=true 2023-04-15 02:56:08,034 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 20e761db5975fb9082f870bf056bd078: 2023-04-15 02:56:08,034 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681527356526.20e761db5975fb9082f870bf056bd078. 2023-04-15 02:56:08,034 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-15 02:56:08,034 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-15 02:56:08,034 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:08,034 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-15 02:56:08,034 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34681,1681527209133' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-15 02:56:08,035 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:08,035 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:56:08,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:08,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:56:08,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:56:08,036 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:56:08,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:56:08,036 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-15 02:56:08,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:56:08,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:56:08,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:08,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:56:08,036 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34681,1681527209133' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-15 02:56:08,037 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2762fce5[Count = 0] remaining members to acquire global barrier 2023-04-15 02:56:08,037 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-15 02:56:08,037 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:56:08,037 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:56:08,037 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:56:08,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:56:08,037 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-15 02:56:08,037 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-15 02:56:08,037 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34681,1681527209133' in zk 2023-04-15 02:56:08,037 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:08,037 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-15 02:56:08,038 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:08,038 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-15 02:56:08,038 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-15 02:56:08,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:08,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:56:08,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:56:08,038 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-15 02:56:08,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:56:08,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:56:08,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:56:08,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:08,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:56:08,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:56:08,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:08,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34681,1681527209133': 2023-04-15 02:56:08,040 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34681,1681527209133' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-15 02:56:08,040 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-15 02:56:08,040 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-15 02:56:08,040 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-15 02:56:08,040 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-15 02:56:08,040 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-15 02:56:08,041 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:56:08,041 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:56:08,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:56:08,041 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:56:08,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:56:08,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:56:08,041 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-15 02:56:08,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:56:08,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:56:08,042 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-15 02:56:08,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-15 02:56:08,042 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:08,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:56:08,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:56:08,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:56:08,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:56:08,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:08,043 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:08,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:56:08,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:56:08,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:08,045 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:08,045 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-15 02:56:08,045 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:56:08,045 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-15 02:56:08,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-15 02:56:08,045 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-15 02:56:08,045 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-15 02:56:08,045 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-15 02:56:08,045 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-15 02:56:08,045 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:56:08,045 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:08,045 DEBUG [Thread-3439] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-15 02:56:08,045 DEBUG [Thread-3439] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-15 02:56:08,045 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:56:08,045 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:56:08,045 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:56:08,045 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-15 02:56:08,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-15 02:56:08,145 DEBUG [Thread-3439] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-15 02:56:08,146 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-15 02:56:08,147 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-15 02:56:08,148 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-15 02:56:08,148 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-15 02:56:08,148 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-15 02:56:08,148 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-15 02:56:08,148 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-15 02:56:08,148 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-15 02:56:08,148 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:56:08,148 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:56:08,149 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-15 02:56:08,149 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-15 02:56:08,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-15 02:56:08,149 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:08,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:56:08,149 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:08,149 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-15 02:56:08,149 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:56:08,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-15 02:56:08,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:56:08,150 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-15 02:56:08,150 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-15 02:56:08,150 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-15 02:56:08,150 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-15 02:56:08,150 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-15 02:56:08,150 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-15 02:56:08,150 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-15 02:56:08,150 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-15 02:56:08,150 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681527356526.20e761db5975fb9082f870bf056bd078. 2023-04-15 02:56:08,150 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681527356526.20e761db5975fb9082f870bf056bd078. started... 2023-04-15 02:56:08,150 INFO [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 20e761db5975fb9082f870bf056bd078 3/3 column families, dataSize=53.67 KB heapSize=160.13 KB 2023-04-15 02:56:08,151 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 20e761db5975fb9082f870bf056bd078, store=A 2023-04-15 02:56:08,151 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-15 02:56:08,151 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 20e761db5975fb9082f870bf056bd078, store=B 2023-04-15 02:56:08,151 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-15 02:56:08,151 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 20e761db5975fb9082f870bf056bd078, store=C 2023-04-15 02:56:08,151 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-15 02:56:08,559 INFO [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230415fd633ad9c04941d1a47e1d7f15f6d9a6 to hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230415fd633ad9c04941d1a47e1d7f15f6d9a6 2023-04-15 02:56:08,563 INFO [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=181, memsize=17.9 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/20e761db5975fb9082f870bf056bd078/.tmp/A/3536d0084b564ccca286193f0c26d057 2023-04-15 02:56:08,672 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34681] regionserver.HRegion(9158): Flush requested on 20e761db5975fb9082f870bf056bd078 2023-04-15 02:56:08,672 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681527356526.20e761db5975fb9082f870bf056bd078. as already flushing 2023-04-15 02:56:08,696 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=20e761db5975fb9082f870bf056bd078, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:56:08,696 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34681] ipc.CallRunner(144): callId: 92 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:48664 deadline: 1681527428692, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=20e761db5975fb9082f870bf056bd078, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:08,696 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=20e761db5975fb9082f870bf056bd078, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:56:08,696 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=20e761db5975fb9082f870bf056bd078, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:56:08,696 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34681] ipc.CallRunner(144): callId: 91 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:48648 deadline: 1681527428692, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=20e761db5975fb9082f870bf056bd078, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:08,696 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34681] ipc.CallRunner(144): callId: 89 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:48666 deadline: 1681527428692, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=20e761db5975fb9082f870bf056bd078, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:08,696 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681527368682, max timestamp=1681527368695Num uniques 859; 2023-04-15 02:56:08,697 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-5] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-15 02:56:08,697 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681527368682, max timestamp=1681527368695Num uniques 859; 2023-04-15 02:56:08,697 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-6] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-15 02:56:08,697 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681527368682, max timestamp=1681527368695Num uniques 859; 2023-04-15 02:56:08,697 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-8] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-15 02:56:08,748 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=20e761db5975fb9082f870bf056bd078, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:56:08,749 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34681] ipc.CallRunner(144): callId: 106 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:48624 deadline: 1681527428746, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=20e761db5975fb9082f870bf056bd078, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:08,799 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=20e761db5975fb9082f870bf056bd078, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:56:08,799 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34681] ipc.CallRunner(144): callId: 91 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:48666 deadline: 1681527428797, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=20e761db5975fb9082f870bf056bd078, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:08,799 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=20e761db5975fb9082f870bf056bd078, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:56:08,799 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34681] ipc.CallRunner(144): callId: 94 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:48664 deadline: 1681527428797, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=20e761db5975fb9082f870bf056bd078, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:08,799 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=20e761db5975fb9082f870bf056bd078, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:56:08,799 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34681] ipc.CallRunner(144): callId: 93 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:48648 deadline: 1681527428797, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=20e761db5975fb9082f870bf056bd078, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:08,851 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=20e761db5975fb9082f870bf056bd078, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:56:08,851 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34681] ipc.CallRunner(144): callId: 108 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:48624 deadline: 1681527428849, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=20e761db5975fb9082f870bf056bd078, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:08,971 INFO [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=181 (bloomFilter=true), to=hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/20e761db5975fb9082f870bf056bd078/.tmp/B/d4b604a3725b405d939d12f478ff880c 2023-04-15 02:56:09,002 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=20e761db5975fb9082f870bf056bd078, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:56:09,002 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34681] ipc.CallRunner(144): callId: 93 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:48666 deadline: 1681527429000, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=20e761db5975fb9082f870bf056bd078, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:09,002 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=20e761db5975fb9082f870bf056bd078, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:56:09,002 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34681] ipc.CallRunner(144): callId: 95 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:48648 deadline: 1681527429000, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=20e761db5975fb9082f870bf056bd078, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:09,003 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=20e761db5975fb9082f870bf056bd078, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:56:09,003 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34681] ipc.CallRunner(144): callId: 96 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:48664 deadline: 1681527429001, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=20e761db5975fb9082f870bf056bd078, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:09,055 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=20e761db5975fb9082f870bf056bd078, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:56:09,055 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34681] ipc.CallRunner(144): callId: 110 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:48624 deadline: 1681527429053, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=20e761db5975fb9082f870bf056bd078, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:09,304 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=20e761db5975fb9082f870bf056bd078, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:56:09,304 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34681] ipc.CallRunner(144): callId: 97 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:48648 deadline: 1681527429303, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=20e761db5975fb9082f870bf056bd078, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:09,307 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=20e761db5975fb9082f870bf056bd078, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:56:09,307 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34681] ipc.CallRunner(144): callId: 98 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:48664 deadline: 1681527429304, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=20e761db5975fb9082f870bf056bd078, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:09,307 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=20e761db5975fb9082f870bf056bd078, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:56:09,307 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34681] ipc.CallRunner(144): callId: 95 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:48666 deadline: 1681527429304, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=20e761db5975fb9082f870bf056bd078, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:09,359 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=20e761db5975fb9082f870bf056bd078, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:56:09,359 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34681] ipc.CallRunner(144): callId: 112 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:48624 deadline: 1681527429358, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=20e761db5975fb9082f870bf056bd078, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:09,379 INFO [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=181 (bloomFilter=true), to=hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/20e761db5975fb9082f870bf056bd078/.tmp/C/8a7b83ae8b904dcda979b2722064752f 2023-04-15 02:56:09,382 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/20e761db5975fb9082f870bf056bd078/.tmp/A/3536d0084b564ccca286193f0c26d057 as hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/20e761db5975fb9082f870bf056bd078/A/3536d0084b564ccca286193f0c26d057 2023-04-15 02:56:09,385 INFO [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/20e761db5975fb9082f870bf056bd078/A/3536d0084b564ccca286193f0c26d057, entries=150, sequenceid=181, filesize=25.3 K 2023-04-15 02:56:09,385 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/20e761db5975fb9082f870bf056bd078/.tmp/B/d4b604a3725b405d939d12f478ff880c as hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/20e761db5975fb9082f870bf056bd078/B/d4b604a3725b405d939d12f478ff880c 2023-04-15 02:56:09,388 INFO [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/20e761db5975fb9082f870bf056bd078/B/d4b604a3725b405d939d12f478ff880c, entries=150, sequenceid=181, filesize=11.8 K 2023-04-15 02:56:09,389 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/20e761db5975fb9082f870bf056bd078/.tmp/C/8a7b83ae8b904dcda979b2722064752f as hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/20e761db5975fb9082f870bf056bd078/C/8a7b83ae8b904dcda979b2722064752f 2023-04-15 02:56:09,391 INFO [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/20e761db5975fb9082f870bf056bd078/C/8a7b83ae8b904dcda979b2722064752f, entries=150, sequenceid=181, filesize=11.8 K 2023-04-15 02:56:09,392 INFO [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=127.47 KB/130530 for 20e761db5975fb9082f870bf056bd078 in 1242ms, sequenceid=181, compaction requested=true 2023-04-15 02:56:09,392 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 20e761db5975fb9082f870bf056bd078: 2023-04-15 02:56:09,392 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681527356526.20e761db5975fb9082f870bf056bd078. 2023-04-15 02:56:09,392 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-15 02:56:09,392 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-15 02:56:09,392 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:09,392 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-15 02:56:09,392 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34681,1681527209133' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-15 02:56:09,393 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:09,393 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:56:09,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:09,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:56:09,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:56:09,393 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:56:09,393 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-15 02:56:09,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:56:09,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:56:09,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:56:09,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:09,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:56:09,394 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34681,1681527209133' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-15 02:56:09,394 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-15 02:56:09,394 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@969893b[Count = 0] remaining members to acquire global barrier 2023-04-15 02:56:09,394 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:56:09,395 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:56:09,395 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:56:09,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:56:09,395 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-15 02:56:09,395 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-15 02:56:09,395 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34681,1681527209133' in zk 2023-04-15 02:56:09,395 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:09,395 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-15 02:56:09,396 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-15 02:56:09,396 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-15 02:56:09,396 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:09,396 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-15 02:56:09,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:09,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:56:09,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:56:09,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:56:09,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:56:09,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:56:09,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:09,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:56:09,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:56:09,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:09,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34681,1681527209133': 2023-04-15 02:56:09,397 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34681,1681527209133' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-15 02:56:09,397 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-15 02:56:09,397 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-15 02:56:09,397 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-15 02:56:09,398 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-15 02:56:09,398 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-15 02:56:09,398 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:56:09,398 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:56:09,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:56:09,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:56:09,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:56:09,398 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:56:09,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:56:09,398 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-15 02:56:09,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:56:09,399 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:09,399 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-15 02:56:09,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-15 02:56:09,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:56:09,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:56:09,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:56:09,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:56:09,400 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:09,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:09,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:56:09,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:56:09,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:09,401 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-15 02:56:09,401 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-15 02:56:09,401 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:09,401 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-15 02:56:09,401 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-15 02:56:09,401 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:56:09,401 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-15 02:56:09,401 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-15 02:56:09,401 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-15 02:56:09,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-15 02:56:09,401 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:56:09,401 DEBUG [Thread-3439] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-15 02:56:09,401 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:09,401 DEBUG [Thread-3439] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-15 02:56:09,401 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-15 02:56:09,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-15 02:56:09,401 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:56:09,401 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:56:09,401 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:56:09,501 DEBUG [Thread-3439] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-15 02:56:09,502 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-15 02:56:09,503 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-15 02:56:09,504 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-15 02:56:09,504 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-15 02:56:09,504 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-15 02:56:09,504 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-15 02:56:09,504 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-15 02:56:09,504 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-15 02:56:09,505 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:56:09,505 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:56:09,505 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:09,505 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-15 02:56:09,505 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-15 02:56:09,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-15 02:56:09,506 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:09,506 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-15 02:56:09,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:56:09,506 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:56:09,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-15 02:56:09,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:56:09,506 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-15 02:56:09,506 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-15 02:56:09,506 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-15 02:56:09,506 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-15 02:56:09,506 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-15 02:56:09,506 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-15 02:56:09,507 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-15 02:56:09,507 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-15 02:56:09,507 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681527356526.20e761db5975fb9082f870bf056bd078. 2023-04-15 02:56:09,507 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681527356526.20e761db5975fb9082f870bf056bd078. started... 2023-04-15 02:56:09,507 INFO [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 20e761db5975fb9082f870bf056bd078 3/3 column families, dataSize=127.47 KB heapSize=365.06 KB 2023-04-15 02:56:09,507 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 20e761db5975fb9082f870bf056bd078, store=A 2023-04-15 02:56:09,507 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-15 02:56:09,507 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 20e761db5975fb9082f870bf056bd078, store=B 2023-04-15 02:56:09,507 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-15 02:56:09,507 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 20e761db5975fb9082f870bf056bd078, store=C 2023-04-15 02:56:09,507 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-15 02:56:09,631 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34681] regionserver.HRegion(9158): Flush requested on 20e761db5975fb9082f870bf056bd078 2023-04-15 02:56:09,631 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681527356526.20e761db5975fb9082f870bf056bd078. as already flushing 2023-04-15 02:56:09,658 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=20e761db5975fb9082f870bf056bd078, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:56:09,659 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34681] ipc.CallRunner(144): callId: 101 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:48586 deadline: 1681527429656, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=20e761db5975fb9082f870bf056bd078, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:09,761 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=20e761db5975fb9082f870bf056bd078, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:56:09,761 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34681] ipc.CallRunner(144): callId: 103 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:48586 deadline: 1681527429759, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=20e761db5975fb9082f870bf056bd078, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:09,808 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=20e761db5975fb9082f870bf056bd078, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:56:09,808 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34681] ipc.CallRunner(144): callId: 99 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:48648 deadline: 1681527429807, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=20e761db5975fb9082f870bf056bd078, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:09,811 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=20e761db5975fb9082f870bf056bd078, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:56:09,811 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34681] ipc.CallRunner(144): callId: 100 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:48664 deadline: 1681527429809, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=20e761db5975fb9082f870bf056bd078, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:09,814 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=20e761db5975fb9082f870bf056bd078, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:56:09,814 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34681] ipc.CallRunner(144): callId: 97 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:48666 deadline: 1681527429812, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=20e761db5975fb9082f870bf056bd078, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:09,864 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=20e761db5975fb9082f870bf056bd078, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:56:09,864 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34681] ipc.CallRunner(144): callId: 114 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:48624 deadline: 1681527429862, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=20e761db5975fb9082f870bf056bd078, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:09,922 INFO [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230415bccd2248d1f64c7ba7ae153b79a138f0 to hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230415bccd2248d1f64c7ba7ae153b79a138f0 2023-04-15 02:56:09,926 INFO [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=217, memsize=42.5 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/20e761db5975fb9082f870bf056bd078/.tmp/A/88fc7b8e881e4037b4ce9aadb3c830c6 2023-04-15 02:56:09,964 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=20e761db5975fb9082f870bf056bd078, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:56:09,964 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34681] ipc.CallRunner(144): callId: 105 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:48586 deadline: 1681527429963, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=20e761db5975fb9082f870bf056bd078, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:10,267 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=20e761db5975fb9082f870bf056bd078, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:56:10,267 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34681] ipc.CallRunner(144): callId: 107 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:48586 deadline: 1681527430265, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=20e761db5975fb9082f870bf056bd078, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:10,334 INFO [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=217 (bloomFilter=true), to=hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/20e761db5975fb9082f870bf056bd078/.tmp/B/d82948cce74045688628e15cec64e1fd 2023-04-15 02:56:10,743 INFO [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=217 (bloomFilter=true), to=hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/20e761db5975fb9082f870bf056bd078/.tmp/C/2157590b841f4c6da8be924763d5cf67 2023-04-15 02:56:10,746 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/20e761db5975fb9082f870bf056bd078/.tmp/A/88fc7b8e881e4037b4ce9aadb3c830c6 as hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/20e761db5975fb9082f870bf056bd078/A/88fc7b8e881e4037b4ce9aadb3c830c6 2023-04-15 02:56:10,751 INFO [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/20e761db5975fb9082f870bf056bd078/A/88fc7b8e881e4037b4ce9aadb3c830c6, entries=150, sequenceid=217, filesize=25.3 K 2023-04-15 02:56:10,752 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/20e761db5975fb9082f870bf056bd078/.tmp/B/d82948cce74045688628e15cec64e1fd as hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/20e761db5975fb9082f870bf056bd078/B/d82948cce74045688628e15cec64e1fd 2023-04-15 02:56:10,755 INFO [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/20e761db5975fb9082f870bf056bd078/B/d82948cce74045688628e15cec64e1fd, entries=150, sequenceid=217, filesize=11.8 K 2023-04-15 02:56:10,755 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/20e761db5975fb9082f870bf056bd078/.tmp/C/2157590b841f4c6da8be924763d5cf67 as hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/20e761db5975fb9082f870bf056bd078/C/2157590b841f4c6da8be924763d5cf67 2023-04-15 02:56:10,758 INFO [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/20e761db5975fb9082f870bf056bd078/C/2157590b841f4c6da8be924763d5cf67, entries=150, sequenceid=217, filesize=11.8 K 2023-04-15 02:56:10,758 INFO [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~127.47 KB/130530, heapSize ~365.02 KB/373776, currentSize=53.67 KB/54960 for 20e761db5975fb9082f870bf056bd078 in 1251ms, sequenceid=217, compaction requested=true 2023-04-15 02:56:10,758 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 20e761db5975fb9082f870bf056bd078: 2023-04-15 02:56:10,758 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681527356526.20e761db5975fb9082f870bf056bd078. 2023-04-15 02:56:10,759 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-15 02:56:10,759 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-15 02:56:10,759 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:10,759 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-15 02:56:10,759 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34681,1681527209133' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-15 02:56:10,760 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:10,760 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:56:10,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:10,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:56:10,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:56:10,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:56:10,760 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:56:10,760 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-15 02:56:10,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:56:10,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:56:10,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:10,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:56:10,761 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34681,1681527209133' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-15 02:56:10,761 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@12873d88[Count = 0] remaining members to acquire global barrier 2023-04-15 02:56:10,761 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-15 02:56:10,761 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:56:10,762 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:56:10,762 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:56:10,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:56:10,762 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-15 02:56:10,762 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-15 02:56:10,762 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34681,1681527209133' in zk 2023-04-15 02:56:10,762 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:10,762 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-15 02:56:10,763 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-15 02:56:10,763 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-15 02:56:10,763 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-15 02:56:10,763 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:10,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:10,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:56:10,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:56:10,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:56:10,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:56:10,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:56:10,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:10,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:56:10,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:56:10,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:10,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34681,1681527209133': 2023-04-15 02:56:10,765 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34681,1681527209133' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-15 02:56:10,765 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-15 02:56:10,765 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-15 02:56:10,765 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-15 02:56:10,765 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-15 02:56:10,765 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-15 02:56:10,766 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:56:10,766 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:56:10,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:56:10,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:56:10,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:56:10,766 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-15 02:56:10,766 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:56:10,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:56:10,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:56:10,767 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-15 02:56:10,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-15 02:56:10,767 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:10,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:56:10,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:56:10,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:56:10,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:56:10,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:10,768 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:10,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:56:10,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:56:10,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:10,770 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:10,770 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:56:10,770 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:56:10,770 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-15 02:56:10,770 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-15 02:56:10,770 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-15 02:56:10,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-15 02:56:10,770 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:10,770 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-15 02:56:10,770 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-15 02:56:10,770 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-15 02:56:10,770 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-15 02:56:10,770 DEBUG [Thread-3439] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-15 02:56:10,771 DEBUG [Thread-3439] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-15 02:56:10,770 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:56:10,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-15 02:56:10,771 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:56:10,771 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:56:10,772 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34681] regionserver.HRegion(9158): Flush requested on 20e761db5975fb9082f870bf056bd078 2023-04-15 02:56:10,772 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 20e761db5975fb9082f870bf056bd078 3/3 column families, dataSize=60.38 KB heapSize=180.05 KB 2023-04-15 02:56:10,772 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 20e761db5975fb9082f870bf056bd078, store=A 2023-04-15 02:56:10,772 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-15 02:56:10,772 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 20e761db5975fb9082f870bf056bd078, store=B 2023-04-15 02:56:10,772 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-15 02:56:10,772 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 20e761db5975fb9082f870bf056bd078, store=C 2023-04-15 02:56:10,773 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-15 02:56:10,820 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=20e761db5975fb9082f870bf056bd078, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:56:10,820 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34681] ipc.CallRunner(144): callId: 103 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:48664 deadline: 1681527430816, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=20e761db5975fb9082f870bf056bd078, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:10,821 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=20e761db5975fb9082f870bf056bd078, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:56:10,821 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34681] ipc.CallRunner(144): callId: 126 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:48586 deadline: 1681527430818, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=20e761db5975fb9082f870bf056bd078, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:10,821 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=20e761db5975fb9082f870bf056bd078, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:56:10,821 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34681] ipc.CallRunner(144): callId: 101 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:48648 deadline: 1681527430818, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=20e761db5975fb9082f870bf056bd078, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:10,821 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=20e761db5975fb9082f870bf056bd078, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:56:10,821 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34681] ipc.CallRunner(144): callId: 99 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:48666 deadline: 1681527430819, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=20e761db5975fb9082f870bf056bd078, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:10,871 DEBUG [Thread-3439] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-15 02:56:10,871 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-15 02:56:10,872 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-15 02:56:10,873 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-15 02:56:10,874 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-15 02:56:10,874 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=20e761db5975fb9082f870bf056bd078, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:56:10,874 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-15 02:56:10,874 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-15 02:56:10,874 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34681] ipc.CallRunner(144): callId: 116 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:48624 deadline: 1681527430872, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=20e761db5975fb9082f870bf056bd078, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:10,874 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-15 02:56:10,874 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-15 02:56:10,874 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:56:10,874 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:56:10,875 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-15 02:56:10,875 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:10,875 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-15 02:56:10,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-15 02:56:10,875 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:10,875 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-15 02:56:10,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:56:10,875 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:56:10,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-15 02:56:10,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:56:10,876 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-15 02:56:10,876 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-15 02:56:10,876 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-15 02:56:10,876 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-15 02:56:10,876 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-15 02:56:10,876 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-15 02:56:10,876 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-15 02:56:10,876 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-15 02:56:10,876 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681527356526.20e761db5975fb9082f870bf056bd078. 2023-04-15 02:56:10,876 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681527356526.20e761db5975fb9082f870bf056bd078. started... 2023-04-15 02:56:10,876 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681527356526.20e761db5975fb9082f870bf056bd078. as already flushing 2023-04-15 02:56:10,876 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681527356526.20e761db5975fb9082f870bf056bd078. 2023-04-15 02:56:10,876 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-15 02:56:10,876 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-15 02:56:10,876 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:10,876 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-15 02:56:10,877 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34681,1681527209133' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-15 02:56:10,877 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:56:10,877 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:10,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:10,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:56:10,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:56:10,877 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:56:10,877 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-15 02:56:10,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:56:10,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:56:10,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:56:10,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:10,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:56:10,879 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34681,1681527209133' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-15 02:56:10,879 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-15 02:56:10,879 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7616affd[Count = 0] remaining members to acquire global barrier 2023-04-15 02:56:10,879 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:56:10,879 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:56:10,879 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:56:10,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:56:10,879 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-15 02:56:10,879 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:10,879 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-15 02:56:10,879 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-15 02:56:10,879 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34681,1681527209133' in zk 2023-04-15 02:56:10,880 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-15 02:56:10,880 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-15 02:56:10,880 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-15 02:56:10,880 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:10,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:10,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:56:10,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:56:10,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:56:10,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:56:10,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:56:10,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:10,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:56:10,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:56:10,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:10,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34681,1681527209133': 2023-04-15 02:56:10,882 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34681,1681527209133' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-15 02:56:10,882 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-15 02:56:10,882 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-15 02:56:10,882 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-15 02:56:10,882 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-15 02:56:10,882 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-15 02:56:10,883 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:56:10,883 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:56:10,883 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:56:10,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:56:10,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:56:10,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:56:10,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:56:10,883 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-15 02:56:10,884 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:10,884 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-15 02:56:10,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-15 02:56:10,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:56:10,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:56:10,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:56:10,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:56:10,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:56:10,885 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:10,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:10,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:56:10,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:56:10,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:10,886 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:10,887 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:56:10,887 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:56:10,887 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-15 02:56:10,887 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:10,887 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-15 02:56:10,887 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-15 02:56:10,887 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:56:10,887 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-15 02:56:10,887 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:56:10,887 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-15 02:56:10,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-15 02:56:10,887 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-15 02:56:10,887 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-15 02:56:10,887 DEBUG [Thread-3439] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-15 02:56:10,887 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:56:10,887 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-15 02:56:10,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-15 02:56:10,887 DEBUG [Thread-3439] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-15 02:56:10,924 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=20e761db5975fb9082f870bf056bd078, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:56:10,924 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34681] ipc.CallRunner(144): callId: 128 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:48586 deadline: 1681527430922, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=20e761db5975fb9082f870bf056bd078, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:10,925 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=20e761db5975fb9082f870bf056bd078, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:56:10,925 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34681] ipc.CallRunner(144): callId: 105 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:48664 deadline: 1681527430922, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=20e761db5975fb9082f870bf056bd078, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:10,987 DEBUG [Thread-3439] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-15 02:56:10,988 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-15 02:56:10,989 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-15 02:56:10,990 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-15 02:56:10,990 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-15 02:56:10,990 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-15 02:56:10,990 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-15 02:56:10,990 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-15 02:56:10,990 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-15 02:56:10,991 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:56:10,991 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:56:10,991 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:10,991 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-15 02:56:10,991 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-15 02:56:10,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-15 02:56:10,992 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:10,992 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-15 02:56:10,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:56:10,992 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:56:10,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-15 02:56:10,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:56:10,992 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-15 02:56:10,992 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-15 02:56:10,992 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-15 02:56:10,992 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-15 02:56:10,992 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-15 02:56:10,993 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-15 02:56:10,993 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-15 02:56:10,993 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-15 02:56:10,993 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681527356526.20e761db5975fb9082f870bf056bd078. 2023-04-15 02:56:10,993 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681527356526.20e761db5975fb9082f870bf056bd078. started... 2023-04-15 02:56:10,993 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681527356526.20e761db5975fb9082f870bf056bd078. as already flushing 2023-04-15 02:56:10,993 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681527356526.20e761db5975fb9082f870bf056bd078. 2023-04-15 02:56:10,993 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-15 02:56:10,993 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-15 02:56:10,993 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:10,993 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-15 02:56:10,993 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34681,1681527209133' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-15 02:56:10,994 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:10,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:10,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:56:10,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:56:10,994 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:56:10,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:56:10,994 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:56:10,994 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-15 02:56:10,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:56:10,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:56:10,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:10,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:56:10,995 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34681,1681527209133' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-15 02:56:10,995 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-15 02:56:10,995 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@46a3e00e[Count = 0] remaining members to acquire global barrier 2023-04-15 02:56:10,995 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:56:10,996 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:56:10,996 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:56:10,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:56:10,996 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-15 02:56:10,996 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-15 02:56:10,996 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34681,1681527209133' in zk 2023-04-15 02:56:10,996 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:10,996 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-15 02:56:10,997 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:10,997 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-15 02:56:10,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:10,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:56:10,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:56:10,997 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-15 02:56:10,997 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-15 02:56:10,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:56:10,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:56:10,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:56:10,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:10,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:56:10,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:56:10,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:11,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34681,1681527209133': 2023-04-15 02:56:11,000 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34681,1681527209133' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-15 02:56:11,000 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-15 02:56:11,000 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-15 02:56:11,000 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-15 02:56:11,000 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-15 02:56:11,000 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-15 02:56:11,000 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:56:11,001 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:56:11,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:56:11,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:56:11,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:56:11,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:56:11,001 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:11,001 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:56:11,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:56:11,001 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-15 02:56:11,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:56:11,002 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-15 02:56:11,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-15 02:56:11,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:56:11,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:56:11,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:56:11,002 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:11,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:11,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:56:11,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:56:11,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:11,003 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:11,004 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:56:11,004 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:56:11,004 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-15 02:56:11,004 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:11,004 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-15 02:56:11,004 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-15 02:56:11,004 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-15 02:56:11,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-15 02:56:11,004 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-15 02:56:11,004 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-15 02:56:11,004 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-15 02:56:11,004 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:56:11,004 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:56:11,004 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:56:11,004 DEBUG [Thread-3439] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-15 02:56:11,004 DEBUG [Thread-3439] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-15 02:56:11,004 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-15 02:56:11,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-15 02:56:11,104 DEBUG [Thread-3439] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-15 02:56:11,105 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-15 02:56:11,106 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-15 02:56:11,107 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-15 02:56:11,107 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-15 02:56:11,107 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-15 02:56:11,107 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-15 02:56:11,107 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-15 02:56:11,107 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-15 02:56:11,107 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:56:11,108 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:56:11,108 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-15 02:56:11,108 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:11,108 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-15 02:56:11,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-15 02:56:11,108 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:11,109 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-15 02:56:11,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:56:11,109 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:56:11,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-15 02:56:11,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:56:11,109 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-15 02:56:11,109 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-15 02:56:11,109 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-15 02:56:11,109 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-15 02:56:11,109 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-15 02:56:11,109 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-15 02:56:11,110 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-15 02:56:11,110 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-15 02:56:11,110 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681527356526.20e761db5975fb9082f870bf056bd078. 2023-04-15 02:56:11,110 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681527356526.20e761db5975fb9082f870bf056bd078. started... 2023-04-15 02:56:11,110 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681527356526.20e761db5975fb9082f870bf056bd078. as already flushing 2023-04-15 02:56:11,110 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681527356526.20e761db5975fb9082f870bf056bd078. 2023-04-15 02:56:11,110 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-15 02:56:11,110 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-15 02:56:11,110 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:11,110 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-15 02:56:11,110 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34681,1681527209133' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-15 02:56:11,111 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:11,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:11,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:56:11,111 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:56:11,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:56:11,111 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:56:11,111 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-15 02:56:11,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:56:11,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:56:11,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:56:11,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:11,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:56:11,112 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34681,1681527209133' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-15 02:56:11,112 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3c689b0[Count = 0] remaining members to acquire global barrier 2023-04-15 02:56:11,112 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-15 02:56:11,112 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:56:11,113 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:56:11,113 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:56:11,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:56:11,113 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-15 02:56:11,113 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-15 02:56:11,113 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34681,1681527209133' in zk 2023-04-15 02:56:11,113 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:11,113 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-15 02:56:11,113 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:11,113 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-15 02:56:11,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:11,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:56:11,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:56:11,113 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-15 02:56:11,114 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-15 02:56:11,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:56:11,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:56:11,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:56:11,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:11,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:56:11,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:56:11,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:11,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34681,1681527209133': 2023-04-15 02:56:11,115 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34681,1681527209133' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-15 02:56:11,115 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-15 02:56:11,115 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-15 02:56:11,115 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-15 02:56:11,115 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-15 02:56:11,115 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-15 02:56:11,117 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:56:11,117 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:56:11,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:56:11,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:56:11,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:56:11,117 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-15 02:56:11,117 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:56:11,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:56:11,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:56:11,117 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:11,117 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-15 02:56:11,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-15 02:56:11,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:56:11,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:56:11,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:56:11,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:56:11,118 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:11,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:11,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:56:11,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:56:11,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:11,120 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:11,120 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:56:11,120 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:56:11,120 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-15 02:56:11,120 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-15 02:56:11,120 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-15 02:56:11,120 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-15 02:56:11,120 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:11,120 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-15 02:56:11,120 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-15 02:56:11,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-15 02:56:11,120 DEBUG [Thread-3439] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-15 02:56:11,120 DEBUG [Thread-3439] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-15 02:56:11,120 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:56:11,120 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:56:11,120 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:56:11,120 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-15 02:56:11,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-15 02:56:11,128 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=20e761db5975fb9082f870bf056bd078, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:56:11,128 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34681] ipc.CallRunner(144): callId: 130 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:48586 deadline: 1681527431126, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=20e761db5975fb9082f870bf056bd078, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:11,128 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=20e761db5975fb9082f870bf056bd078, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:56:11,128 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34681] ipc.CallRunner(144): callId: 107 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:48664 deadline: 1681527431126, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=20e761db5975fb9082f870bf056bd078, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:11,182 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230415b0f6cf38abe84579bf13c51e746e826c to hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230415b0f6cf38abe84579bf13c51e746e826c 2023-04-15 02:56:11,220 DEBUG [Thread-3439] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-15 02:56:11,221 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-15 02:56:11,222 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-15 02:56:11,223 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-15 02:56:11,223 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-15 02:56:11,223 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-15 02:56:11,223 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-15 02:56:11,223 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-15 02:56:11,223 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-15 02:56:11,223 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:56:11,223 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:56:11,224 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-15 02:56:11,224 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-15 02:56:11,224 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:11,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-15 02:56:11,224 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:11,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:56:11,224 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-15 02:56:11,225 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:56:11,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-15 02:56:11,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:56:11,225 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-15 02:56:11,225 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-15 02:56:11,225 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-15 02:56:11,225 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-15 02:56:11,225 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-15 02:56:11,225 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-15 02:56:11,225 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-15 02:56:11,225 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-15 02:56:11,225 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681527356526.20e761db5975fb9082f870bf056bd078. 2023-04-15 02:56:11,225 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681527356526.20e761db5975fb9082f870bf056bd078. started... 2023-04-15 02:56:11,225 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681527356526.20e761db5975fb9082f870bf056bd078. as already flushing 2023-04-15 02:56:11,225 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681527356526.20e761db5975fb9082f870bf056bd078. 2023-04-15 02:56:11,225 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-15 02:56:11,226 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-15 02:56:11,226 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:11,226 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-15 02:56:11,226 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34681,1681527209133' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-15 02:56:11,226 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:11,226 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:56:11,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:11,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:56:11,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:56:11,227 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:56:11,227 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-15 02:56:11,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:56:11,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:56:11,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:56:11,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:11,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:56:11,228 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34681,1681527209133' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-15 02:56:11,228 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-15 02:56:11,228 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@28068f3d[Count = 0] remaining members to acquire global barrier 2023-04-15 02:56:11,228 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:56:11,229 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:56:11,229 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:56:11,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:56:11,229 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-15 02:56:11,229 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-15 02:56:11,229 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:11,229 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-15 02:56:11,229 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34681,1681527209133' in zk 2023-04-15 02:56:11,229 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:11,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:11,229 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-15 02:56:11,230 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-15 02:56:11,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:56:11,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:56:11,230 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-15 02:56:11,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:56:11,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:56:11,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:56:11,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:11,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:56:11,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:56:11,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:11,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34681,1681527209133': 2023-04-15 02:56:11,231 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34681,1681527209133' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-15 02:56:11,231 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-15 02:56:11,231 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-15 02:56:11,231 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-15 02:56:11,232 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-15 02:56:11,232 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-15 02:56:11,233 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:56:11,233 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-15 02:56:11,233 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:56:11,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:56:11,233 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-15 02:56:11,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-15 02:56:11,233 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:56:11,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:56:11,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:56:11,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:56:11,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:56:11,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:56:11,233 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:11,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:56:11,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:56:11,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:56:11,234 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:11,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:11,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:56:11,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:56:11,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:11,235 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-15 02:56:11,236 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-15 02:56:11,236 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-15 02:56:11,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-15 02:56:11,236 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-15 02:56:11,236 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:11,236 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-15 02:56:11,236 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-15 02:56:11,236 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:56:11,236 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-15 02:56:11,236 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:56:11,236 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-15 02:56:11,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-15 02:56:11,236 DEBUG [Thread-3439] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-15 02:56:11,236 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:11,236 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:56:11,236 DEBUG [Thread-3439] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-15 02:56:11,236 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:56:11,236 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:56:11,336 DEBUG [Thread-3439] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-15 02:56:11,337 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-15 02:56:11,338 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-15 02:56:11,339 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-15 02:56:11,339 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-15 02:56:11,339 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-15 02:56:11,339 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-15 02:56:11,339 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-15 02:56:11,339 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-15 02:56:11,340 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:56:11,340 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:56:11,340 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-15 02:56:11,340 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:11,340 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-15 02:56:11,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-15 02:56:11,341 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:11,341 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-15 02:56:11,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:56:11,341 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:56:11,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-15 02:56:11,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:56:11,341 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-15 02:56:11,342 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-15 02:56:11,342 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-15 02:56:11,342 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-15 02:56:11,342 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-15 02:56:11,342 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-15 02:56:11,342 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-15 02:56:11,342 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-15 02:56:11,342 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681527356526.20e761db5975fb9082f870bf056bd078. 2023-04-15 02:56:11,342 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681527356526.20e761db5975fb9082f870bf056bd078. started... 2023-04-15 02:56:11,342 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681527356526.20e761db5975fb9082f870bf056bd078. as already flushing 2023-04-15 02:56:11,342 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681527356526.20e761db5975fb9082f870bf056bd078. 2023-04-15 02:56:11,342 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-15 02:56:11,342 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-15 02:56:11,342 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:11,342 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-15 02:56:11,342 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34681,1681527209133' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-15 02:56:11,343 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:11,343 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:56:11,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:11,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:56:11,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:56:11,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:56:11,344 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:56:11,344 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-15 02:56:11,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:56:11,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:56:11,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:11,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:56:11,345 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34681,1681527209133' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-15 02:56:11,345 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-15 02:56:11,345 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6b2ca487[Count = 0] remaining members to acquire global barrier 2023-04-15 02:56:11,345 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:56:11,346 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:56:11,346 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:56:11,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:56:11,346 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-15 02:56:11,346 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:11,346 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-15 02:56:11,346 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-15 02:56:11,346 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34681,1681527209133' in zk 2023-04-15 02:56:11,346 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-15 02:56:11,347 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:11,347 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-15 02:56:11,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:11,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:56:11,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:56:11,347 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-15 02:56:11,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:56:11,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:56:11,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:56:11,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:11,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:56:11,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:56:11,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:11,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34681,1681527209133': 2023-04-15 02:56:11,349 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34681,1681527209133' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-15 02:56:11,349 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-15 02:56:11,349 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-15 02:56:11,349 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-15 02:56:11,349 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-15 02:56:11,349 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-15 02:56:11,350 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:56:11,350 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:56:11,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:56:11,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:56:11,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:56:11,350 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:56:11,350 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-15 02:56:11,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:56:11,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:56:11,350 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:11,350 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-15 02:56:11,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-15 02:56:11,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:56:11,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:56:11,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:56:11,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:56:11,351 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:11,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:11,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:56:11,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:56:11,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:11,353 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:11,353 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:56:11,353 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-15 02:56:11,353 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-15 02:56:11,353 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-15 02:56:11,353 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-15 02:56:11,353 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-15 02:56:11,353 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-15 02:56:11,353 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-15 02:56:11,353 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:56:11,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-15 02:56:11,353 DEBUG [Thread-3439] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-15 02:56:11,353 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:11,353 DEBUG [Thread-3439] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-15 02:56:11,353 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:56:11,353 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:56:11,353 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:56:11,353 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-15 02:56:11,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-15 02:56:11,430 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=20e761db5975fb9082f870bf056bd078, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:56:11,430 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34681] ipc.CallRunner(144): callId: 132 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:48586 deadline: 1681527431429, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=20e761db5975fb9082f870bf056bd078, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:11,431 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=20e761db5975fb9082f870bf056bd078, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:56:11,431 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34681] ipc.CallRunner(144): callId: 109 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:48664 deadline: 1681527431430, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=20e761db5975fb9082f870bf056bd078, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:11,453 DEBUG [Thread-3439] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-15 02:56:11,454 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-15 02:56:11,455 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-15 02:56:11,456 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-15 02:56:11,456 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-15 02:56:11,456 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-15 02:56:11,456 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-15 02:56:11,456 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-15 02:56:11,456 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-15 02:56:11,456 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:56:11,456 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:56:11,457 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-15 02:56:11,457 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:11,457 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-15 02:56:11,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-15 02:56:11,457 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:11,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:56:11,457 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-15 02:56:11,457 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:56:11,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-15 02:56:11,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:56:11,458 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-15 02:56:11,458 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-15 02:56:11,458 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-15 02:56:11,458 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-15 02:56:11,458 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-15 02:56:11,458 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-15 02:56:11,458 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-15 02:56:11,458 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-15 02:56:11,458 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681527356526.20e761db5975fb9082f870bf056bd078. 2023-04-15 02:56:11,458 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681527356526.20e761db5975fb9082f870bf056bd078. started... 2023-04-15 02:56:11,458 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681527356526.20e761db5975fb9082f870bf056bd078. as already flushing 2023-04-15 02:56:11,458 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681527356526.20e761db5975fb9082f870bf056bd078. 2023-04-15 02:56:11,458 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-15 02:56:11,458 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-15 02:56:11,459 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:11,459 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-15 02:56:11,459 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34681,1681527209133' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-15 02:56:11,459 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:11,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:11,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:56:11,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:56:11,459 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:56:11,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:56:11,460 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:56:11,460 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-15 02:56:11,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:56:11,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:56:11,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:11,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:56:11,460 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34681,1681527209133' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-15 02:56:11,461 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-15 02:56:11,461 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@32569a7c[Count = 0] remaining members to acquire global barrier 2023-04-15 02:56:11,461 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:56:11,461 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:56:11,461 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:56:11,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:56:11,461 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:11,461 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-15 02:56:11,461 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-15 02:56:11,461 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-15 02:56:11,461 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34681,1681527209133' in zk 2023-04-15 02:56:11,462 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-15 02:56:11,462 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-15 02:56:11,462 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:11,462 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-15 02:56:11,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:11,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:56:11,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:56:11,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:56:11,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:56:11,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:56:11,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:11,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:56:11,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:56:11,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:11,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34681,1681527209133': 2023-04-15 02:56:11,464 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34681,1681527209133' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-15 02:56:11,464 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-15 02:56:11,464 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-15 02:56:11,464 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-15 02:56:11,464 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-15 02:56:11,464 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-15 02:56:11,465 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:56:11,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:56:11,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:56:11,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:56:11,465 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:56:11,465 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-15 02:56:11,465 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:56:11,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:56:11,465 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-15 02:56:11,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:56:11,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-15 02:56:11,465 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:11,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:56:11,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:56:11,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:56:11,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:56:11,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:11,466 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:11,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:56:11,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:56:11,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:11,468 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-15 02:56:11,468 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-15 02:56:11,468 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-15 02:56:11,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-15 02:56:11,468 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-15 02:56:11,468 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:11,468 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:56:11,468 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:56:11,468 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-15 02:56:11,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-15 02:56:11,468 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-15 02:56:11,468 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-15 02:56:11,468 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:11,468 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:56:11,468 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:56:11,469 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:56:11,469 DEBUG [Thread-3439] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-15 02:56:11,469 DEBUG [Thread-3439] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-15 02:56:11,569 DEBUG [Thread-3439] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-15 02:56:11,569 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-15 02:56:11,570 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-15 02:56:11,571 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-15 02:56:11,571 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-15 02:56:11,571 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-15 02:56:11,571 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-15 02:56:11,571 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-15 02:56:11,571 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-15 02:56:11,572 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:56:11,572 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:56:11,572 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-15 02:56:11,572 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:11,572 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-15 02:56:11,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-15 02:56:11,573 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:11,573 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-15 02:56:11,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:56:11,573 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:56:11,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-15 02:56:11,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:56:11,573 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-15 02:56:11,573 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-15 02:56:11,573 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-15 02:56:11,573 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-15 02:56:11,573 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-15 02:56:11,574 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-15 02:56:11,574 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-15 02:56:11,574 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-15 02:56:11,574 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681527356526.20e761db5975fb9082f870bf056bd078. 2023-04-15 02:56:11,574 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681527356526.20e761db5975fb9082f870bf056bd078. started... 2023-04-15 02:56:11,574 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681527356526.20e761db5975fb9082f870bf056bd078. as already flushing 2023-04-15 02:56:11,574 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681527356526.20e761db5975fb9082f870bf056bd078. 2023-04-15 02:56:11,574 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-15 02:56:11,574 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-15 02:56:11,574 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:11,574 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-15 02:56:11,574 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34681,1681527209133' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-15 02:56:11,575 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:56:11,575 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:11,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:11,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:56:11,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:56:11,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:56:11,575 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:56:11,575 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-15 02:56:11,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:56:11,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:56:11,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:11,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:56:11,576 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34681,1681527209133' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-15 02:56:11,576 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-15 02:56:11,576 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@56a0e8f[Count = 0] remaining members to acquire global barrier 2023-04-15 02:56:11,576 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:56:11,577 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:56:11,577 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:56:11,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:56:11,577 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-15 02:56:11,577 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-15 02:56:11,577 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34681,1681527209133' in zk 2023-04-15 02:56:11,577 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:11,577 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-15 02:56:11,578 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:11,578 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-15 02:56:11,578 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-15 02:56:11,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:11,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:56:11,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:56:11,578 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-15 02:56:11,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:56:11,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:56:11,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:56:11,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:11,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:56:11,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:56:11,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:11,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34681,1681527209133': 2023-04-15 02:56:11,579 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34681,1681527209133' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-15 02:56:11,579 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-15 02:56:11,579 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-15 02:56:11,579 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-15 02:56:11,579 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-15 02:56:11,579 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-15 02:56:11,580 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:56:11,580 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:56:11,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:56:11,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:56:11,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:56:11,580 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:56:11,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:56:11,580 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-15 02:56:11,580 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:11,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:56:11,580 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-15 02:56:11,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-15 02:56:11,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:56:11,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:56:11,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:56:11,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:56:11,581 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:11,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:11,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:56:11,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:56:11,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:11,583 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:11,583 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-15 02:56:11,583 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:56:11,583 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-15 02:56:11,583 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-15 02:56:11,583 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-15 02:56:11,583 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-15 02:56:11,583 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-15 02:56:11,583 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-15 02:56:11,583 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:56:11,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-15 02:56:11,583 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:11,583 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:56:11,583 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:56:11,583 DEBUG [Thread-3439] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-15 02:56:11,583 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:56:11,583 DEBUG [Thread-3439] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-15 02:56:11,583 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-15 02:56:11,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-15 02:56:11,586 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=229, memsize=20.1 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/20e761db5975fb9082f870bf056bd078/.tmp/A/ad3786b6e53e4004823e008fe6273198 2023-04-15 02:56:11,683 DEBUG [Thread-3439] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-15 02:56:11,684 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-15 02:56:11,685 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-15 02:56:11,686 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-15 02:56:11,686 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-15 02:56:11,686 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-15 02:56:11,686 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-15 02:56:11,686 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-15 02:56:11,686 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-15 02:56:11,686 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:56:11,686 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:56:11,687 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:11,687 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-15 02:56:11,687 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-15 02:56:11,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-15 02:56:11,687 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:11,687 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-15 02:56:11,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:56:11,687 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:56:11,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-15 02:56:11,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:56:11,688 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-15 02:56:11,688 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-15 02:56:11,688 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-15 02:56:11,688 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-15 02:56:11,688 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-15 02:56:11,688 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-15 02:56:11,688 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-15 02:56:11,688 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-15 02:56:11,688 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681527356526.20e761db5975fb9082f870bf056bd078. 2023-04-15 02:56:11,688 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681527356526.20e761db5975fb9082f870bf056bd078. started... 2023-04-15 02:56:11,688 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681527356526.20e761db5975fb9082f870bf056bd078. as already flushing 2023-04-15 02:56:11,688 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681527356526.20e761db5975fb9082f870bf056bd078. 2023-04-15 02:56:11,688 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-15 02:56:11,688 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-15 02:56:11,688 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:11,688 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-15 02:56:11,688 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34681,1681527209133' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-15 02:56:11,689 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:11,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:11,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:56:11,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:56:11,689 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:56:11,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:56:11,689 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:56:11,689 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-15 02:56:11,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:56:11,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:56:11,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:11,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:56:11,690 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34681,1681527209133' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-15 02:56:11,690 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-15 02:56:11,690 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@19091678[Count = 0] remaining members to acquire global barrier 2023-04-15 02:56:11,690 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:56:11,691 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:56:11,691 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:56:11,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:56:11,691 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-15 02:56:11,691 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-15 02:56:11,691 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:11,691 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-15 02:56:11,691 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34681,1681527209133' in zk 2023-04-15 02:56:11,692 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:11,692 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-15 02:56:11,692 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-15 02:56:11,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:11,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:56:11,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:56:11,692 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-15 02:56:11,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:56:11,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:56:11,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:56:11,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:11,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:56:11,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:56:11,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:11,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34681,1681527209133': 2023-04-15 02:56:11,693 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34681,1681527209133' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-15 02:56:11,693 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-15 02:56:11,693 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-15 02:56:11,693 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-15 02:56:11,693 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-15 02:56:11,693 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-15 02:56:11,694 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:56:11,694 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:56:11,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:56:11,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:56:11,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:56:11,694 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-15 02:56:11,694 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:56:11,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:56:11,694 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:11,694 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-15 02:56:11,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:56:11,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-15 02:56:11,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:56:11,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:56:11,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:56:11,695 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:11,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:56:11,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:11,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:56:11,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:56:11,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:11,696 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:11,696 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-15 02:56:11,696 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-15 02:56:11,696 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:56:11,696 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-15 02:56:11,696 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-15 02:56:11,696 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-15 02:56:11,696 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-15 02:56:11,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-15 02:56:11,697 DEBUG [Thread-3439] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-15 02:56:11,697 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-15 02:56:11,696 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:56:11,697 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:11,697 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-15 02:56:11,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-15 02:56:11,697 DEBUG [Thread-3439] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-15 02:56:11,697 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:56:11,697 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:56:11,697 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:56:11,797 DEBUG [Thread-3439] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-15 02:56:11,797 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-15 02:56:11,799 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-15 02:56:11,800 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-15 02:56:11,800 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-15 02:56:11,800 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-15 02:56:11,800 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-15 02:56:11,800 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-15 02:56:11,800 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-15 02:56:11,802 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:56:11,802 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:56:11,803 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-15 02:56:11,803 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:11,803 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-15 02:56:11,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-15 02:56:11,803 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:11,803 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-15 02:56:11,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:56:11,804 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:56:11,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-15 02:56:11,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:56:11,804 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-15 02:56:11,804 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-15 02:56:11,804 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-15 02:56:11,804 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-15 02:56:11,804 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-15 02:56:11,804 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-15 02:56:11,804 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-15 02:56:11,804 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681527356526.20e761db5975fb9082f870bf056bd078. 2023-04-15 02:56:11,804 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681527356526.20e761db5975fb9082f870bf056bd078. started... 2023-04-15 02:56:11,804 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681527356526.20e761db5975fb9082f870bf056bd078. as already flushing 2023-04-15 02:56:11,804 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681527356526.20e761db5975fb9082f870bf056bd078. 2023-04-15 02:56:11,805 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-15 02:56:11,805 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-15 02:56:11,805 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:11,805 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-15 02:56:11,805 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34681,1681527209133' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-15 02:56:11,805 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:56:11,805 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:11,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:11,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:56:11,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:56:11,806 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:56:11,806 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-15 02:56:11,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:56:11,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:56:11,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:56:11,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:11,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:56:11,807 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34681,1681527209133' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-15 02:56:11,807 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-15 02:56:11,807 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@170f8142[Count = 0] remaining members to acquire global barrier 2023-04-15 02:56:11,807 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:56:11,808 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:56:11,808 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:56:11,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:56:11,808 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-15 02:56:11,808 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-15 02:56:11,808 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34681,1681527209133' in zk 2023-04-15 02:56:11,808 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:11,808 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-15 02:56:11,809 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:11,809 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-15 02:56:11,809 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-15 02:56:11,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:11,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:56:11,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:56:11,809 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-15 02:56:11,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:56:11,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:56:11,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:56:11,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:11,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:56:11,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:56:11,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:11,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34681,1681527209133': 2023-04-15 02:56:11,810 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34681,1681527209133' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-15 02:56:11,810 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-15 02:56:11,811 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-15 02:56:11,811 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-15 02:56:11,811 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-15 02:56:11,811 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-15 02:56:11,811 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:56:11,811 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:56:11,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:56:11,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:56:11,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:56:11,811 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:56:11,811 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-15 02:56:11,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:56:11,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:56:11,812 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:11,812 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-15 02:56:11,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-15 02:56:11,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:56:11,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:56:11,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:56:11,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:56:11,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:11,813 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:11,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:56:11,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:56:11,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:11,815 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:11,815 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:56:11,815 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-15 02:56:11,815 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:56:11,815 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-15 02:56:11,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-15 02:56:11,815 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-15 02:56:11,816 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:11,816 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:56:11,816 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-15 02:56:11,816 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-15 02:56:11,816 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-15 02:56:11,816 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-15 02:56:11,816 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-15 02:56:11,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-15 02:56:11,816 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:56:11,816 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:56:11,816 DEBUG [Thread-3439] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-15 02:56:11,816 DEBUG [Thread-3439] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-15 02:56:11,916 DEBUG [Thread-3439] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-15 02:56:11,916 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-15 02:56:11,918 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-15 02:56:11,918 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-15 02:56:11,919 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-15 02:56:11,919 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-15 02:56:11,919 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-15 02:56:11,919 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-15 02:56:11,919 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-15 02:56:11,919 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:56:11,919 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:56:11,920 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:11,920 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-15 02:56:11,920 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-15 02:56:11,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-15 02:56:11,920 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:11,920 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-15 02:56:11,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:56:11,920 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:56:11,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-15 02:56:11,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:56:11,921 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-15 02:56:11,921 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-15 02:56:11,921 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-15 02:56:11,921 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-15 02:56:11,921 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-15 02:56:11,921 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-15 02:56:11,921 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-15 02:56:11,921 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-15 02:56:11,921 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681527356526.20e761db5975fb9082f870bf056bd078. 2023-04-15 02:56:11,921 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681527356526.20e761db5975fb9082f870bf056bd078. started... 2023-04-15 02:56:11,921 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681527356526.20e761db5975fb9082f870bf056bd078. as already flushing 2023-04-15 02:56:11,921 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681527356526.20e761db5975fb9082f870bf056bd078. 2023-04-15 02:56:11,921 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-15 02:56:11,921 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-15 02:56:11,921 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:11,921 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-15 02:56:11,921 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34681,1681527209133' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-15 02:56:11,922 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:11,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:11,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:56:11,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:56:11,922 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:56:11,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:56:11,923 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:56:11,923 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-15 02:56:11,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:56:11,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:56:11,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:11,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:56:11,923 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34681,1681527209133' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-15 02:56:11,923 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@118c7ca5[Count = 0] remaining members to acquire global barrier 2023-04-15 02:56:11,923 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-15 02:56:11,923 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:56:11,924 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:56:11,924 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:56:11,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:56:11,924 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-15 02:56:11,924 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-15 02:56:11,924 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34681,1681527209133' in zk 2023-04-15 02:56:11,924 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:11,924 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-15 02:56:11,925 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-15 02:56:11,925 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-15 02:56:11,925 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:11,925 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-15 02:56:11,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:11,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:56:11,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:56:11,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:56:11,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:56:11,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:56:11,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:11,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:56:11,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:56:11,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:11,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34681,1681527209133': 2023-04-15 02:56:11,927 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34681,1681527209133' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-15 02:56:11,927 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-15 02:56:11,927 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-15 02:56:11,927 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-15 02:56:11,927 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-15 02:56:11,927 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-15 02:56:11,927 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:56:11,928 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:56:11,928 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-15 02:56:11,928 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:56:11,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:56:11,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:56:11,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:56:11,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:56:11,928 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-15 02:56:11,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-15 02:56:11,928 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:11,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:56:11,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:56:11,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:56:11,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:56:11,929 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:11,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:56:11,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:11,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:56:11,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:56:11,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:11,930 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-15 02:56:11,930 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-15 02:56:11,930 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:11,930 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-15 02:56:11,930 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-15 02:56:11,930 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-15 02:56:11,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-15 02:56:11,930 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-15 02:56:11,930 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:56:11,930 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-15 02:56:11,930 DEBUG [Thread-3439] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-15 02:56:11,930 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:56:11,931 DEBUG [Thread-3439] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-15 02:56:11,931 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:11,931 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-15 02:56:11,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-15 02:56:11,931 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:56:11,931 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:56:11,931 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:56:11,934 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=20e761db5975fb9082f870bf056bd078, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:56:11,934 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34681] ipc.CallRunner(144): callId: 111 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:48664 deadline: 1681527431932, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=20e761db5975fb9082f870bf056bd078, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:11,935 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=20e761db5975fb9082f870bf056bd078, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:56:11,935 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34681] ipc.CallRunner(144): callId: 134 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:48586 deadline: 1681527431934, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=20e761db5975fb9082f870bf056bd078, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:11,994 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=229 (bloomFilter=true), to=hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/20e761db5975fb9082f870bf056bd078/.tmp/B/579e24eeec284cf2ae86663ebdf3335f 2023-04-15 02:56:12,000 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=229 (bloomFilter=true), to=hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/20e761db5975fb9082f870bf056bd078/.tmp/C/50e1c4ae2a13474aa591f904b786e56e 2023-04-15 02:56:12,004 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/20e761db5975fb9082f870bf056bd078/.tmp/A/ad3786b6e53e4004823e008fe6273198 as hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/20e761db5975fb9082f870bf056bd078/A/ad3786b6e53e4004823e008fe6273198 2023-04-15 02:56:12,008 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/20e761db5975fb9082f870bf056bd078/A/ad3786b6e53e4004823e008fe6273198, entries=200, sequenceid=229, filesize=32.1 K 2023-04-15 02:56:12,008 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/20e761db5975fb9082f870bf056bd078/.tmp/B/579e24eeec284cf2ae86663ebdf3335f as hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/20e761db5975fb9082f870bf056bd078/B/579e24eeec284cf2ae86663ebdf3335f 2023-04-15 02:56:12,011 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/20e761db5975fb9082f870bf056bd078/B/579e24eeec284cf2ae86663ebdf3335f, entries=150, sequenceid=229, filesize=11.8 K 2023-04-15 02:56:12,013 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/20e761db5975fb9082f870bf056bd078/.tmp/C/50e1c4ae2a13474aa591f904b786e56e as hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/20e761db5975fb9082f870bf056bd078/C/50e1c4ae2a13474aa591f904b786e56e 2023-04-15 02:56:12,020 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/20e761db5975fb9082f870bf056bd078/C/50e1c4ae2a13474aa591f904b786e56e, entries=150, sequenceid=229, filesize=11.8 K 2023-04-15 02:56:12,020 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~60.38 KB/61830, heapSize ~180 KB/184320, currentSize=114.05 KB/116790 for 20e761db5975fb9082f870bf056bd078 in 1248ms, sequenceid=229, compaction requested=true 2023-04-15 02:56:12,020 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 20e761db5975fb9082f870bf056bd078: 2023-04-15 02:56:12,021 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-15 02:56:12,021 DEBUG [RS:0;jenkins-hbase20:34681-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 6 store files, 0 compacting, 6 eligible, 16 blocking 2023-04-15 02:56:12,021 DEBUG [RS:0;jenkins-hbase20:34681-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 6 store files, 0 compacting, 6 eligible, 16 blocking 2023-04-15 02:56:12,021 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-15 02:56:12,021 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-15 02:56:12,022 DEBUG [RS:0;jenkins-hbase20:34681-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 6 files of size 162406 starting at candidate #0 after considering 10 permutations with 10 in ratio 2023-04-15 02:56:12,022 DEBUG [RS:0;jenkins-hbase20:34681-shortCompactions-0] regionserver.HStore(1912): 20e761db5975fb9082f870bf056bd078/A is initiating minor compaction (all files) 2023-04-15 02:56:12,022 DEBUG [RS:0;jenkins-hbase20:34681-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 6 files of size 72256 starting at candidate #0 after considering 10 permutations with 10 in ratio 2023-04-15 02:56:12,022 INFO [RS:0;jenkins-hbase20:34681-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 20e761db5975fb9082f870bf056bd078/A in TestAcidGuarantees,,1681527356526.20e761db5975fb9082f870bf056bd078. 2023-04-15 02:56:12,022 DEBUG [RS:0;jenkins-hbase20:34681-longCompactions-0] regionserver.HStore(1912): 20e761db5975fb9082f870bf056bd078/B is initiating minor compaction (all files) 2023-04-15 02:56:12,022 INFO [RS:0;jenkins-hbase20:34681-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/20e761db5975fb9082f870bf056bd078/A/1839c025181b488a9f7e760e76b34ff6, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/20e761db5975fb9082f870bf056bd078/A/835c5281097746cb9ba12cc89ec8adbe, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/20e761db5975fb9082f870bf056bd078/A/168e7843b8c9470f857fedbb03a0eb8c, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/20e761db5975fb9082f870bf056bd078/A/3536d0084b564ccca286193f0c26d057, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/20e761db5975fb9082f870bf056bd078/A/88fc7b8e881e4037b4ce9aadb3c830c6, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/20e761db5975fb9082f870bf056bd078/A/ad3786b6e53e4004823e008fe6273198] into tmpdir=hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/20e761db5975fb9082f870bf056bd078/.tmp, totalSize=158.6 K 2023-04-15 02:56:12,022 INFO [RS:0;jenkins-hbase20:34681-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 20e761db5975fb9082f870bf056bd078/B in TestAcidGuarantees,,1681527356526.20e761db5975fb9082f870bf056bd078. 2023-04-15 02:56:12,023 INFO [RS:0;jenkins-hbase20:34681-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/20e761db5975fb9082f870bf056bd078/B/f092b0e0f6b44cb280f40a09566e7c43, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/20e761db5975fb9082f870bf056bd078/B/13265909e92d49d793b4cf103df530a3, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/20e761db5975fb9082f870bf056bd078/B/06e500fe23174a8896b3d0f9dd3a4863, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/20e761db5975fb9082f870bf056bd078/B/d4b604a3725b405d939d12f478ff880c, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/20e761db5975fb9082f870bf056bd078/B/d82948cce74045688628e15cec64e1fd, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/20e761db5975fb9082f870bf056bd078/B/579e24eeec284cf2ae86663ebdf3335f] into tmpdir=hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/20e761db5975fb9082f870bf056bd078/.tmp, totalSize=70.6 K 2023-04-15 02:56:12,023 DEBUG [RS:0;jenkins-hbase20:34681-shortCompactions-0] compactions.Compactor(207): Compacting 1839c025181b488a9f7e760e76b34ff6, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=120, earliestPutTs=1681527363236 2023-04-15 02:56:12,023 DEBUG [RS:0;jenkins-hbase20:34681-longCompactions-0] compactions.Compactor(207): Compacting f092b0e0f6b44cb280f40a09566e7c43, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=120, earliestPutTs=1681527363236 2023-04-15 02:56:12,023 DEBUG [RS:0;jenkins-hbase20:34681-shortCompactions-0] compactions.Compactor(207): Compacting 835c5281097746cb9ba12cc89ec8adbe, keycount=150, bloomtype=ROW, size=25.2 K, encoding=NONE, compression=NONE, seqNum=132, earliestPutTs=1681527363276 2023-04-15 02:56:12,023 DEBUG [RS:0;jenkins-hbase20:34681-longCompactions-0] compactions.Compactor(207): Compacting 13265909e92d49d793b4cf103df530a3, keycount=150, bloomtype=ROW, size=11.7 K, encoding=NONE, compression=NONE, seqNum=132, earliestPutTs=1681527363276 2023-04-15 02:56:12,024 DEBUG [RS:0;jenkins-hbase20:34681-shortCompactions-0] compactions.Compactor(207): Compacting 168e7843b8c9470f857fedbb03a0eb8c, keycount=150, bloomtype=ROW, size=25.3 K, encoding=NONE, compression=NONE, seqNum=170, earliestPutTs=1681527365461 2023-04-15 02:56:12,024 DEBUG [RS:0;jenkins-hbase20:34681-longCompactions-0] compactions.Compactor(207): Compacting 06e500fe23174a8896b3d0f9dd3a4863, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=170, earliestPutTs=1681527365461 2023-04-15 02:56:12,024 DEBUG [RS:0;jenkins-hbase20:34681-shortCompactions-0] compactions.Compactor(207): Compacting 3536d0084b564ccca286193f0c26d057, keycount=150, bloomtype=ROW, size=25.3 K, encoding=NONE, compression=NONE, seqNum=181, earliestPutTs=1681527366547 2023-04-15 02:56:12,024 DEBUG [RS:0;jenkins-hbase20:34681-longCompactions-0] compactions.Compactor(207): Compacting d4b604a3725b405d939d12f478ff880c, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=181, earliestPutTs=1681527366547 2023-04-15 02:56:12,024 DEBUG [RS:0;jenkins-hbase20:34681-longCompactions-0] compactions.Compactor(207): Compacting d82948cce74045688628e15cec64e1fd, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=217, earliestPutTs=1681527368738 2023-04-15 02:56:12,024 DEBUG [RS:0;jenkins-hbase20:34681-shortCompactions-0] compactions.Compactor(207): Compacting 88fc7b8e881e4037b4ce9aadb3c830c6, keycount=150, bloomtype=ROW, size=25.3 K, encoding=NONE, compression=NONE, seqNum=217, earliestPutTs=1681527368738 2023-04-15 02:56:12,024 DEBUG [RS:0;jenkins-hbase20:34681-longCompactions-0] compactions.Compactor(207): Compacting 579e24eeec284cf2ae86663ebdf3335f, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=229, earliestPutTs=1681527369643 2023-04-15 02:56:12,025 DEBUG [RS:0;jenkins-hbase20:34681-shortCompactions-0] compactions.Compactor(207): Compacting ad3786b6e53e4004823e008fe6273198, keycount=200, bloomtype=ROW, size=32.1 K, encoding=NONE, compression=NONE, seqNum=229, earliestPutTs=1681527369643 2023-04-15 02:56:12,031 DEBUG [Thread-3439] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-15 02:56:12,031 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-15 02:56:12,033 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-15 02:56:12,046 INFO [RS:0;jenkins-hbase20:34681-longCompactions-0] throttle.PressureAwareThroughputController(145): 20e761db5975fb9082f870bf056bd078#B#compaction#479 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-15 02:56:12,047 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-15 02:56:12,047 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-15 02:56:12,047 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-15 02:56:12,047 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-15 02:56:12,047 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-15 02:56:12,047 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-15 02:56:12,048 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:56:12,048 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:56:12,048 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-15 02:56:12,049 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:12,049 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-15 02:56:12,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-15 02:56:12,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:56:12,049 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:12,049 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-15 02:56:12,049 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:56:12,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-15 02:56:12,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:56:12,050 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-15 02:56:12,056 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-15 02:56:12,056 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-15 02:56:12,056 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-15 02:56:12,056 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-15 02:56:12,056 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-15 02:56:12,066 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-15 02:56:12,066 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-15 02:56:12,066 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681527356526.20e761db5975fb9082f870bf056bd078. 2023-04-15 02:56:12,066 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681527356526.20e761db5975fb9082f870bf056bd078. started... 2023-04-15 02:56:12,066 INFO [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 20e761db5975fb9082f870bf056bd078 3/3 column families, dataSize=114.05 KB heapSize=339.42 KB 2023-04-15 02:56:12,066 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 20e761db5975fb9082f870bf056bd078, store=A 2023-04-15 02:56:12,066 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-15 02:56:12,066 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 20e761db5975fb9082f870bf056bd078, store=B 2023-04-15 02:56:12,066 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-15 02:56:12,066 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 20e761db5975fb9082f870bf056bd078, store=C 2023-04-15 02:56:12,066 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-15 02:56:12,067 INFO [RS:0;jenkins-hbase20:34681-shortCompactions-0] throttle.PressureAwareThroughputController(145): 20e761db5975fb9082f870bf056bd078#A#compaction#480 average throughput is 0.99 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-15 02:56:12,118 INFO [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023041518ea87b6f4df457d9bda1542999cb119 to hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023041518ea87b6f4df457d9bda1542999cb119 2023-04-15 02:56:12,128 INFO [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=249, memsize=38.0 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/20e761db5975fb9082f870bf056bd078/.tmp/A/8c4dc0ae326943b4ae95dc85a0dbc4e5 2023-04-15 02:56:12,136 INFO [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=249 (bloomFilter=true), to=hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/20e761db5975fb9082f870bf056bd078/.tmp/B/a48b06a3343d435b86de97231507762d 2023-04-15 02:56:12,143 INFO [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=249 (bloomFilter=true), to=hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/20e761db5975fb9082f870bf056bd078/.tmp/C/df0ecae983e54117bd14be37866047fe 2023-04-15 02:56:12,147 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/20e761db5975fb9082f870bf056bd078/.tmp/A/8c4dc0ae326943b4ae95dc85a0dbc4e5 as hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/20e761db5975fb9082f870bf056bd078/A/8c4dc0ae326943b4ae95dc85a0dbc4e5 2023-04-15 02:56:12,150 INFO [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/20e761db5975fb9082f870bf056bd078/A/8c4dc0ae326943b4ae95dc85a0dbc4e5, entries=150, sequenceid=249, filesize=25.3 K 2023-04-15 02:56:12,150 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/20e761db5975fb9082f870bf056bd078/.tmp/B/a48b06a3343d435b86de97231507762d as hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/20e761db5975fb9082f870bf056bd078/B/a48b06a3343d435b86de97231507762d 2023-04-15 02:56:12,153 INFO [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/20e761db5975fb9082f870bf056bd078/B/a48b06a3343d435b86de97231507762d, entries=150, sequenceid=249, filesize=11.8 K 2023-04-15 02:56:12,154 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/20e761db5975fb9082f870bf056bd078/.tmp/C/df0ecae983e54117bd14be37866047fe as hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/20e761db5975fb9082f870bf056bd078/C/df0ecae983e54117bd14be37866047fe 2023-04-15 02:56:12,157 INFO [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/20e761db5975fb9082f870bf056bd078/C/df0ecae983e54117bd14be37866047fe, entries=150, sequenceid=249, filesize=11.8 K 2023-04-15 02:56:12,158 INFO [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~114.05 KB/116790, heapSize ~339.38 KB/347520, currentSize=0 B/0 for 20e761db5975fb9082f870bf056bd078 in 92ms, sequenceid=249, compaction requested=true 2023-04-15 02:56:12,158 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 20e761db5975fb9082f870bf056bd078: 2023-04-15 02:56:12,158 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681527356526.20e761db5975fb9082f870bf056bd078. 2023-04-15 02:56:12,158 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-15 02:56:12,158 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-15 02:56:12,158 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:12,158 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-15 02:56:12,158 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34681,1681527209133' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-15 02:56:12,159 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:56:12,159 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:12,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:12,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:56:12,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:56:12,159 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:56:12,159 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-15 02:56:12,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:56:12,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:56:12,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:56:12,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:12,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:56:12,161 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34681,1681527209133' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-15 02:56:12,161 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@44c7166d[Count = 0] remaining members to acquire global barrier 2023-04-15 02:56:12,161 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-15 02:56:12,161 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:56:12,161 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:56:12,161 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:56:12,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:56:12,161 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-15 02:56:12,162 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-15 02:56:12,162 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34681,1681527209133' in zk 2023-04-15 02:56:12,162 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:12,162 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-15 02:56:12,162 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:12,162 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-15 02:56:12,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:12,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:56:12,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:56:12,162 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-15 02:56:12,162 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-15 02:56:12,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:56:12,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:56:12,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:56:12,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:12,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:56:12,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:56:12,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:12,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34681,1681527209133': 2023-04-15 02:56:12,164 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34681,1681527209133' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-15 02:56:12,164 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-15 02:56:12,165 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-15 02:56:12,165 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-15 02:56:12,165 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-15 02:56:12,165 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-15 02:56:12,166 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:56:12,166 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:56:12,166 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:56:12,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:56:12,166 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-15 02:56:12,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:56:12,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:56:12,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:56:12,166 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-15 02:56:12,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-15 02:56:12,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:56:12,166 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:12,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:56:12,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:56:12,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:56:12,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:56:12,167 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:12,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:12,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:56:12,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:56:12,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:12,169 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:12,169 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:56:12,169 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-15 02:56:12,169 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-15 02:56:12,169 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-15 02:56:12,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-15 02:56:12,169 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:56:12,169 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:12,169 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:56:12,169 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:56:12,169 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:56:12,169 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-15 02:56:12,169 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-15 02:56:12,169 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-15 02:56:12,169 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-15 02:56:12,169 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-15 02:56:12,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-15 02:56:12,169 DEBUG [Thread-3439] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-15 02:56:12,169 DEBUG [Thread-3439] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-15 02:56:12,270 DEBUG [Thread-3439] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-15 02:56:12,270 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-15 02:56:12,271 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-15 02:56:12,274 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-15 02:56:12,274 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-15 02:56:12,274 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-15 02:56:12,274 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-15 02:56:12,274 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-15 02:56:12,274 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-15 02:56:12,276 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:56:12,276 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:56:12,277 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-15 02:56:12,277 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-15 02:56:12,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-15 02:56:12,277 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:12,277 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:12,277 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-15 02:56:12,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:56:12,278 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:56:12,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-15 02:56:12,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:56:12,278 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-15 02:56:12,279 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-15 02:56:12,279 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-15 02:56:12,279 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-15 02:56:12,279 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-15 02:56:12,279 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-15 02:56:12,279 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-15 02:56:12,279 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-15 02:56:12,279 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681527356526.20e761db5975fb9082f870bf056bd078. 2023-04-15 02:56:12,279 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681527356526.20e761db5975fb9082f870bf056bd078. started... 2023-04-15 02:56:12,279 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 20e761db5975fb9082f870bf056bd078: 2023-04-15 02:56:12,279 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681527356526.20e761db5975fb9082f870bf056bd078. 2023-04-15 02:56:12,279 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-15 02:56:12,279 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-15 02:56:12,279 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:12,279 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-15 02:56:12,279 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34681,1681527209133' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-15 02:56:12,280 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:56:12,280 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:12,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:12,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:56:12,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:56:12,280 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:56:12,280 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-15 02:56:12,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:56:12,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:56:12,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:56:12,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:12,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:56:12,282 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34681,1681527209133' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-15 02:56:12,282 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@152c725b[Count = 0] remaining members to acquire global barrier 2023-04-15 02:56:12,282 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-15 02:56:12,282 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:56:12,282 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:56:12,282 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:56:12,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:56:12,283 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-15 02:56:12,283 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-15 02:56:12,283 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:12,283 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-15 02:56:12,283 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34681,1681527209133' in zk 2023-04-15 02:56:12,283 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-15 02:56:12,283 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:12,283 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-15 02:56:12,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:12,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:56:12,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:56:12,284 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-15 02:56:12,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:56:12,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:56:12,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:56:12,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:12,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:56:12,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:56:12,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:12,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34681,1681527209133': 2023-04-15 02:56:12,286 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34681,1681527209133' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-15 02:56:12,286 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-15 02:56:12,286 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-15 02:56:12,286 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-15 02:56:12,286 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-15 02:56:12,286 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-15 02:56:12,287 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:56:12,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:56:12,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:56:12,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:56:12,287 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:56:12,287 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-15 02:56:12,287 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:56:12,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:56:12,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:56:12,287 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-15 02:56:12,287 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:12,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-15 02:56:12,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:56:12,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:56:12,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:56:12,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:56:12,288 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:12,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:12,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:56:12,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:56:12,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:12,290 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:12,290 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-15 02:56:12,290 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:56:12,290 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-15 02:56:12,290 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-15 02:56:12,290 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-15 02:56:12,290 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-15 02:56:12,290 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:56:12,290 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-15 02:56:12,290 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-15 02:56:12,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-15 02:56:12,290 DEBUG [Thread-3439] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-15 02:56:12,290 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:12,290 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:56:12,290 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:56:12,290 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:56:12,290 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-15 02:56:12,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-15 02:56:12,290 DEBUG [Thread-3439] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-15 02:56:12,391 DEBUG [Thread-3439] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-15 02:56:12,391 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-15 02:56:12,392 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-15 02:56:12,393 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-15 02:56:12,393 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-15 02:56:12,393 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-15 02:56:12,393 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-15 02:56:12,393 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-15 02:56:12,393 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-15 02:56:12,394 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:56:12,394 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:56:12,395 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-15 02:56:12,395 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:12,395 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-15 02:56:12,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-15 02:56:12,395 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:12,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:56:12,395 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-15 02:56:12,395 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:56:12,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-15 02:56:12,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:56:12,395 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-15 02:56:12,396 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-15 02:56:12,396 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-15 02:56:12,396 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-15 02:56:12,396 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-15 02:56:12,396 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-15 02:56:12,396 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-15 02:56:12,396 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-15 02:56:12,396 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681527356526.20e761db5975fb9082f870bf056bd078. 2023-04-15 02:56:12,396 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681527356526.20e761db5975fb9082f870bf056bd078. started... 2023-04-15 02:56:12,396 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 20e761db5975fb9082f870bf056bd078: 2023-04-15 02:56:12,396 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681527356526.20e761db5975fb9082f870bf056bd078. 2023-04-15 02:56:12,397 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-15 02:56:12,397 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-15 02:56:12,397 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:12,397 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-15 02:56:12,397 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34681,1681527209133' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-15 02:56:12,398 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:12,398 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:56:12,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:12,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:56:12,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:56:12,399 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:56:12,399 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-15 02:56:12,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:56:12,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:56:12,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:56:12,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:12,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:56:12,400 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34681,1681527209133' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-15 02:56:12,400 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-15 02:56:12,400 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7e42e42d[Count = 0] remaining members to acquire global barrier 2023-04-15 02:56:12,400 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:56:12,400 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:56:12,400 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:56:12,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:56:12,400 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-15 02:56:12,400 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-15 02:56:12,400 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:12,400 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-15 02:56:12,400 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34681,1681527209133' in zk 2023-04-15 02:56:12,401 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:12,401 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-15 02:56:12,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:12,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:56:12,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:56:12,401 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-15 02:56:12,401 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-15 02:56:12,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:56:12,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:56:12,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:56:12,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:12,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:56:12,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:56:12,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:12,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34681,1681527209133': 2023-04-15 02:56:12,403 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34681,1681527209133' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-15 02:56:12,403 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-15 02:56:12,403 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-15 02:56:12,403 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-15 02:56:12,403 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-15 02:56:12,403 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-15 02:56:12,404 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:56:12,404 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:56:12,404 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:56:12,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:56:12,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:56:12,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:56:12,404 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-15 02:56:12,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:56:12,405 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-15 02:56:12,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-15 02:56:12,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:56:12,405 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:12,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:56:12,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:56:12,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:56:12,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:56:12,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:12,406 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:12,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:56:12,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:56:12,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:12,408 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:12,408 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:56:12,408 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-15 02:56:12,408 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-15 02:56:12,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-15 02:56:12,408 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:56:12,408 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-15 02:56:12,408 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-15 02:56:12,408 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-15 02:56:12,408 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-15 02:56:12,408 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-15 02:56:12,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-15 02:56:12,408 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:12,408 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:56:12,408 DEBUG [Thread-3439] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-15 02:56:12,408 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:56:12,408 DEBUG [Thread-3439] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-15 02:56:12,408 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:56:12,478 DEBUG [RS:0;jenkins-hbase20:34681-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/20e761db5975fb9082f870bf056bd078/.tmp/B/a7e7f41bd36a4aefbcf65317d8aa8ab8 as hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/20e761db5975fb9082f870bf056bd078/B/a7e7f41bd36a4aefbcf65317d8aa8ab8 2023-04-15 02:56:12,481 INFO [RS:0;jenkins-hbase20:34681-longCompactions-0] regionserver.HStore(1652): Completed compaction of 6 (all) file(s) in 20e761db5975fb9082f870bf056bd078/B of 20e761db5975fb9082f870bf056bd078 into a7e7f41bd36a4aefbcf65317d8aa8ab8(size=12.2 K), total size for store is 23.9 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-15 02:56:12,481 DEBUG [RS:0;jenkins-hbase20:34681-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 20e761db5975fb9082f870bf056bd078: 2023-04-15 02:56:12,481 INFO [RS:0;jenkins-hbase20:34681-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681527356526.20e761db5975fb9082f870bf056bd078., storeName=20e761db5975fb9082f870bf056bd078/B, priority=10, startTime=1681527372021; duration=0sec 2023-04-15 02:56:12,481 DEBUG [RS:0;jenkins-hbase20:34681-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-15 02:56:12,481 DEBUG [RS:0;jenkins-hbase20:34681-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 7 store files, 0 compacting, 7 eligible, 16 blocking 2023-04-15 02:56:12,483 DEBUG [RS:0;jenkins-hbase20:34681-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 7 files of size 84292 starting at candidate #0 after considering 15 permutations with 15 in ratio 2023-04-15 02:56:12,483 DEBUG [RS:0;jenkins-hbase20:34681-longCompactions-0] regionserver.HStore(1912): 20e761db5975fb9082f870bf056bd078/C is initiating minor compaction (all files) 2023-04-15 02:56:12,483 INFO [RS:0;jenkins-hbase20:34681-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 20e761db5975fb9082f870bf056bd078/C in TestAcidGuarantees,,1681527356526.20e761db5975fb9082f870bf056bd078. 2023-04-15 02:56:12,483 INFO [RS:0;jenkins-hbase20:34681-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/20e761db5975fb9082f870bf056bd078/C/138a4607009640ba9d4821f3f7e6e712, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/20e761db5975fb9082f870bf056bd078/C/def7bce379f0445f9fdfd2b2f011416a, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/20e761db5975fb9082f870bf056bd078/C/633488e8312e4475b90ebf1ca8f775dd, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/20e761db5975fb9082f870bf056bd078/C/8a7b83ae8b904dcda979b2722064752f, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/20e761db5975fb9082f870bf056bd078/C/2157590b841f4c6da8be924763d5cf67, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/20e761db5975fb9082f870bf056bd078/C/50e1c4ae2a13474aa591f904b786e56e, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/20e761db5975fb9082f870bf056bd078/C/df0ecae983e54117bd14be37866047fe] into tmpdir=hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/20e761db5975fb9082f870bf056bd078/.tmp, totalSize=82.3 K 2023-04-15 02:56:12,484 DEBUG [RS:0;jenkins-hbase20:34681-longCompactions-0] compactions.Compactor(207): Compacting 138a4607009640ba9d4821f3f7e6e712, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=120, earliestPutTs=1681527363236 2023-04-15 02:56:12,484 DEBUG [RS:0;jenkins-hbase20:34681-longCompactions-0] compactions.Compactor(207): Compacting def7bce379f0445f9fdfd2b2f011416a, keycount=150, bloomtype=ROW, size=11.7 K, encoding=NONE, compression=NONE, seqNum=132, earliestPutTs=1681527363276 2023-04-15 02:56:12,484 DEBUG [RS:0;jenkins-hbase20:34681-longCompactions-0] compactions.Compactor(207): Compacting 633488e8312e4475b90ebf1ca8f775dd, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=170, earliestPutTs=1681527365461 2023-04-15 02:56:12,485 DEBUG [RS:0;jenkins-hbase20:34681-longCompactions-0] compactions.Compactor(207): Compacting 8a7b83ae8b904dcda979b2722064752f, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=181, earliestPutTs=1681527366547 2023-04-15 02:56:12,485 DEBUG [RS:0;jenkins-hbase20:34681-longCompactions-0] compactions.Compactor(207): Compacting 2157590b841f4c6da8be924763d5cf67, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=217, earliestPutTs=1681527368738 2023-04-15 02:56:12,485 DEBUG [RS:0;jenkins-hbase20:34681-longCompactions-0] compactions.Compactor(207): Compacting 50e1c4ae2a13474aa591f904b786e56e, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=229, earliestPutTs=1681527369643 2023-04-15 02:56:12,486 DEBUG [RS:0;jenkins-hbase20:34681-longCompactions-0] compactions.Compactor(207): Compacting df0ecae983e54117bd14be37866047fe, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=249, earliestPutTs=1681527370809 2023-04-15 02:56:12,493 INFO [RS:0;jenkins-hbase20:34681-longCompactions-0] throttle.PressureAwareThroughputController(145): 20e761db5975fb9082f870bf056bd078#C#compaction#484 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-15 02:56:12,502 DEBUG [RS:0;jenkins-hbase20:34681-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/20e761db5975fb9082f870bf056bd078/.tmp/C/7e2ca3a28dc249e9a1d84c90efb5603f as hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/20e761db5975fb9082f870bf056bd078/C/7e2ca3a28dc249e9a1d84c90efb5603f 2023-04-15 02:56:12,516 DEBUG [Thread-3439] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-15 02:56:12,516 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-15 02:56:12,516 INFO [RS:0;jenkins-hbase20:34681-longCompactions-0] regionserver.HStore(1652): Completed compaction of 7 (all) file(s) in 20e761db5975fb9082f870bf056bd078/C of 20e761db5975fb9082f870bf056bd078 into 7e2ca3a28dc249e9a1d84c90efb5603f(size=12.2 K), total size for store is 12.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-15 02:56:12,516 DEBUG [RS:0;jenkins-hbase20:34681-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 20e761db5975fb9082f870bf056bd078: 2023-04-15 02:56:12,517 INFO [RS:0;jenkins-hbase20:34681-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681527356526.20e761db5975fb9082f870bf056bd078., storeName=20e761db5975fb9082f870bf056bd078/C, priority=9, startTime=1681527372021; duration=0sec 2023-04-15 02:56:12,517 DEBUG [RS:0;jenkins-hbase20:34681-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-15 02:56:12,518 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-15 02:56:12,526 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-15 02:56:12,526 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-15 02:56:12,526 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-15 02:56:12,526 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-15 02:56:12,526 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-15 02:56:12,526 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-15 02:56:12,526 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:56:12,527 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:56:12,527 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-15 02:56:12,527 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:12,527 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-15 02:56:12,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-15 02:56:12,528 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:12,528 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-15 02:56:12,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:56:12,528 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:56:12,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-15 02:56:12,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:56:12,529 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-15 02:56:12,530 DEBUG [RS:0;jenkins-hbase20:34681-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/20e761db5975fb9082f870bf056bd078/.tmp/A/a59ed8a7c4bd40a6879c548ffb1870d2 as hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/20e761db5975fb9082f870bf056bd078/A/a59ed8a7c4bd40a6879c548ffb1870d2 2023-04-15 02:56:12,534 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-15 02:56:12,536 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-15 02:56:12,536 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-15 02:56:12,536 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-15 02:56:12,536 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-15 02:56:12,536 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-15 02:56:12,536 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681527356526.20e761db5975fb9082f870bf056bd078. 2023-04-15 02:56:12,537 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681527356526.20e761db5975fb9082f870bf056bd078. started... 2023-04-15 02:56:12,537 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 20e761db5975fb9082f870bf056bd078: 2023-04-15 02:56:12,537 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681527356526.20e761db5975fb9082f870bf056bd078. 2023-04-15 02:56:12,537 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-15 02:56:12,537 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-15 02:56:12,537 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:12,537 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-15 02:56:12,537 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34681,1681527209133' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-15 02:56:12,537 INFO [RS:0;jenkins-hbase20:34681-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 6 (all) file(s) in 20e761db5975fb9082f870bf056bd078/A of 20e761db5975fb9082f870bf056bd078 into a59ed8a7c4bd40a6879c548ffb1870d2(size=25.7 K), total size for store is 51.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-15 02:56:12,537 DEBUG [RS:0;jenkins-hbase20:34681-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 20e761db5975fb9082f870bf056bd078: 2023-04-15 02:56:12,538 INFO [RS:0;jenkins-hbase20:34681-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681527356526.20e761db5975fb9082f870bf056bd078., storeName=20e761db5975fb9082f870bf056bd078/A, priority=10, startTime=1681527372021; duration=0sec 2023-04-15 02:56:12,538 DEBUG [RS:0;jenkins-hbase20:34681-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-15 02:56:12,538 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:12,539 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:56:12,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:12,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:56:12,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:56:12,539 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:56:12,539 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-15 02:56:12,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:56:12,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:56:12,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:56:12,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:12,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:56:12,540 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34681,1681527209133' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-15 02:56:12,540 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@c59d614[Count = 0] remaining members to acquire global barrier 2023-04-15 02:56:12,540 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-15 02:56:12,540 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:56:12,540 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:56:12,540 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:56:12,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:56:12,541 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-15 02:56:12,541 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:12,541 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-15 02:56:12,541 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-15 02:56:12,541 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34681,1681527209133' in zk 2023-04-15 02:56:12,541 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:12,541 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-15 02:56:12,541 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-15 02:56:12,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:12,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:56:12,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:56:12,542 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-15 02:56:12,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:56:12,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:56:12,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:56:12,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:12,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:56:12,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:56:12,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:12,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34681,1681527209133': 2023-04-15 02:56:12,543 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34681,1681527209133' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-15 02:56:12,543 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-15 02:56:12,543 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-15 02:56:12,543 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-15 02:56:12,543 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-15 02:56:12,544 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-15 02:56:12,544 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:56:12,544 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-15 02:56:12,544 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:56:12,544 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:56:12,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:56:12,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:56:12,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:56:12,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:56:12,545 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-15 02:56:12,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-15 02:56:12,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:56:12,545 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:12,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:56:12,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:56:12,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:56:12,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:56:12,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:12,546 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:12,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:56:12,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:56:12,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:12,548 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:12,548 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:56:12,548 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:56:12,548 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:12,548 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:56:12,548 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-15 02:56:12,548 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-15 02:56:12,548 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-15 02:56:12,548 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-15 02:56:12,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-15 02:56:12,548 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-15 02:56:12,548 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-15 02:56:12,548 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:56:12,548 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:56:12,548 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-15 02:56:12,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-15 02:56:12,548 DEBUG [Thread-3439] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-15 02:56:12,548 DEBUG [Thread-3439] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-15 02:56:12,648 DEBUG [Thread-3439] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-15 02:56:12,649 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-15 02:56:12,650 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-15 02:56:12,651 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-15 02:56:12,651 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-15 02:56:12,651 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-15 02:56:12,651 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-15 02:56:12,651 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-15 02:56:12,651 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-15 02:56:12,651 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:56:12,652 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:56:12,652 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:12,652 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-15 02:56:12,652 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-15 02:56:12,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-15 02:56:12,653 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:12,653 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-15 02:56:12,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:56:12,653 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:56:12,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-15 02:56:12,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:56:12,653 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-15 02:56:12,654 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-15 02:56:12,654 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-15 02:56:12,654 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-15 02:56:12,654 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-15 02:56:12,654 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-15 02:56:12,658 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-15 02:56:12,658 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-15 02:56:12,658 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681527356526.20e761db5975fb9082f870bf056bd078. 2023-04-15 02:56:12,658 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681527356526.20e761db5975fb9082f870bf056bd078. started... 2023-04-15 02:56:12,658 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 20e761db5975fb9082f870bf056bd078: 2023-04-15 02:56:12,658 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681527356526.20e761db5975fb9082f870bf056bd078. 2023-04-15 02:56:12,658 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-15 02:56:12,658 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-15 02:56:12,658 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:12,658 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-15 02:56:12,658 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34681,1681527209133' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-15 02:56:12,662 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:12,662 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:56:12,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:12,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:56:12,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:56:12,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:56:12,662 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:56:12,663 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-15 02:56:12,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:56:12,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:56:12,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:12,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:56:12,664 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34681,1681527209133' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-15 02:56:12,664 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7051dd8c[Count = 0] remaining members to acquire global barrier 2023-04-15 02:56:12,664 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-15 02:56:12,664 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:56:12,664 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:56:12,664 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:56:12,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:56:12,665 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-15 02:56:12,665 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-15 02:56:12,665 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:12,665 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-15 02:56:12,665 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34681,1681527209133' in zk 2023-04-15 02:56:12,666 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:12,666 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-15 02:56:12,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:12,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:56:12,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:56:12,666 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-15 02:56:12,666 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-15 02:56:12,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:56:12,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:56:12,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:56:12,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:12,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:56:12,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:56:12,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:12,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34681,1681527209133': 2023-04-15 02:56:12,668 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34681,1681527209133' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-15 02:56:12,668 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-15 02:56:12,668 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-15 02:56:12,668 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-15 02:56:12,668 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-15 02:56:12,668 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-15 02:56:12,669 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:56:12,669 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:56:12,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:56:12,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:56:12,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:56:12,669 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-15 02:56:12,669 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:56:12,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:56:12,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:56:12,670 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-15 02:56:12,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-15 02:56:12,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:56:12,670 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:12,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:56:12,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:56:12,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:56:12,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:12,671 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:12,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:56:12,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:56:12,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:12,673 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:12,673 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-15 02:56:12,673 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:56:12,673 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-15 02:56:12,673 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:56:12,673 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:12,673 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:56:12,673 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:56:12,673 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:56:12,673 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-15 02:56:12,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-15 02:56:12,673 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-15 02:56:12,673 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-15 02:56:12,673 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-15 02:56:12,674 DEBUG [Thread-3439] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-15 02:56:12,674 DEBUG [Thread-3439] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-15 02:56:12,676 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-15 02:56:12,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-15 02:56:12,774 DEBUG [Thread-3439] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-15 02:56:12,774 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-15 02:56:12,776 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-15 02:56:12,777 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-15 02:56:12,777 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-15 02:56:12,777 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-15 02:56:12,777 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-15 02:56:12,777 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-15 02:56:12,777 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-15 02:56:12,777 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:56:12,777 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:56:12,778 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-15 02:56:12,778 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:12,778 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-15 02:56:12,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-15 02:56:12,778 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:12,778 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-15 02:56:12,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:56:12,779 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:56:12,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-15 02:56:12,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:56:12,779 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-15 02:56:12,779 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-15 02:56:12,779 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-15 02:56:12,779 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-15 02:56:12,779 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-15 02:56:12,779 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-15 02:56:12,786 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-15 02:56:12,786 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-15 02:56:12,786 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681527356526.20e761db5975fb9082f870bf056bd078. 2023-04-15 02:56:12,786 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681527356526.20e761db5975fb9082f870bf056bd078. started... 2023-04-15 02:56:12,786 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 20e761db5975fb9082f870bf056bd078: 2023-04-15 02:56:12,786 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681527356526.20e761db5975fb9082f870bf056bd078. 2023-04-15 02:56:12,786 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-15 02:56:12,786 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-15 02:56:12,786 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:12,787 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-15 02:56:12,787 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34681,1681527209133' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-15 02:56:12,795 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:12,796 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:56:12,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:12,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:56:12,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:56:12,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:56:12,798 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:56:12,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:56:12,798 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-15 02:56:12,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:56:12,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:12,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:56:12,800 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34681,1681527209133' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-15 02:56:12,800 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@699c777[Count = 0] remaining members to acquire global barrier 2023-04-15 02:56:12,800 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-15 02:56:12,800 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:56:12,804 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:56:12,804 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:56:12,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:56:12,804 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-15 02:56:12,805 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-15 02:56:12,805 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34681,1681527209133' in zk 2023-04-15 02:56:12,810 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:12,810 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-15 02:56:12,814 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-15 02:56:12,814 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-15 02:56:12,814 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:12,814 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-15 02:56:12,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:12,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:56:12,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:56:12,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:56:12,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:56:12,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:56:12,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:12,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:56:12,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:56:12,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:12,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34681,1681527209133': 2023-04-15 02:56:12,816 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34681,1681527209133' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-15 02:56:12,816 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-15 02:56:12,816 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-15 02:56:12,816 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-15 02:56:12,816 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-15 02:56:12,816 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-15 02:56:12,817 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:56:12,818 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:56:12,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:56:12,818 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:56:12,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:56:12,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:56:12,818 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-15 02:56:12,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:56:12,818 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-15 02:56:12,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-15 02:56:12,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:56:12,818 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:12,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:56:12,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:56:12,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:56:12,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:56:12,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:12,820 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:12,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:56:12,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:56:12,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:12,822 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-15 02:56:12,822 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-15 02:56:12,822 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:12,822 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-15 02:56:12,822 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-15 02:56:12,822 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-15 02:56:12,822 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-15 02:56:12,823 DEBUG [Thread-3439] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-15 02:56:12,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-15 02:56:12,822 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-15 02:56:12,822 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:56:12,823 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:56:12,823 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:12,823 DEBUG [Thread-3439] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-15 02:56:12,823 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-15 02:56:12,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-15 02:56:12,823 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:56:12,823 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:56:12,823 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:56:12,875 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34681] regionserver.HRegion(9158): Flush requested on 20e761db5975fb9082f870bf056bd078 2023-04-15 02:56:12,875 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 20e761db5975fb9082f870bf056bd078 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-15 02:56:12,876 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 20e761db5975fb9082f870bf056bd078, store=A 2023-04-15 02:56:12,876 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-15 02:56:12,876 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 20e761db5975fb9082f870bf056bd078, store=B 2023-04-15 02:56:12,877 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-15 02:56:12,877 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 20e761db5975fb9082f870bf056bd078, store=C 2023-04-15 02:56:12,877 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-15 02:56:12,904 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=20e761db5975fb9082f870bf056bd078, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:56:12,904 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34681] ipc.CallRunner(144): callId: 121 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:48624 deadline: 1681527432901, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=20e761db5975fb9082f870bf056bd078, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:12,904 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=20e761db5975fb9082f870bf056bd078, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:56:12,904 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34681] ipc.CallRunner(144): callId: 113 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:48666 deadline: 1681527432903, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=20e761db5975fb9082f870bf056bd078, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:12,904 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=20e761db5975fb9082f870bf056bd078, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:56:12,904 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34681] ipc.CallRunner(144): callId: 114 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:48648 deadline: 1681527432903, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=20e761db5975fb9082f870bf056bd078, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:12,905 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681527372896, max timestamp=1681527372902Num uniques 859; 2023-04-15 02:56:12,905 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-4] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-15 02:56:12,905 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681527372896, max timestamp=1681527372902Num uniques 859; 2023-04-15 02:56:12,905 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-1] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-15 02:56:12,907 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681527372896, max timestamp=1681527372902Num uniques 859; 2023-04-15 02:56:12,907 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-2] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-15 02:56:12,907 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230415789f7fdd58984ddba60fade421b9c646 to hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230415789f7fdd58984ddba60fade421b9c646 2023-04-15 02:56:12,920 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=268, memsize=17.9 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/20e761db5975fb9082f870bf056bd078/.tmp/A/7ff283029d7748108529cd72184d2d13 2023-04-15 02:56:12,923 DEBUG [Thread-3439] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-15 02:56:12,924 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-15 02:56:12,927 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-15 02:56:12,928 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-15 02:56:12,929 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-15 02:56:12,929 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-15 02:56:12,929 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-15 02:56:12,929 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-15 02:56:12,929 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-15 02:56:12,929 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:56:12,929 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:56:12,930 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-15 02:56:12,930 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:12,930 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-15 02:56:12,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-15 02:56:12,930 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:12,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:56:12,930 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-15 02:56:12,931 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:56:12,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-15 02:56:12,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:56:12,932 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-15 02:56:12,946 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-15 02:56:12,946 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-15 02:56:12,946 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-15 02:56:12,946 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-15 02:56:12,946 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-15 02:56:12,950 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-15 02:56:12,950 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-15 02:56:12,951 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681527356526.20e761db5975fb9082f870bf056bd078. 2023-04-15 02:56:12,951 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681527356526.20e761db5975fb9082f870bf056bd078. started... 2023-04-15 02:56:12,951 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681527356526.20e761db5975fb9082f870bf056bd078. as already flushing 2023-04-15 02:56:12,951 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681527356526.20e761db5975fb9082f870bf056bd078. 2023-04-15 02:56:12,951 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-15 02:56:12,951 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-15 02:56:12,951 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:12,951 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-15 02:56:12,951 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34681,1681527209133' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-15 02:56:12,952 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:12,952 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:56:12,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:12,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:56:12,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:56:12,952 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=268 (bloomFilter=true), to=hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/20e761db5975fb9082f870bf056bd078/.tmp/B/82721bd5833e4862be0e0abd0b57cde5 2023-04-15 02:56:12,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:56:12,953 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:56:12,953 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-15 02:56:12,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:56:12,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:56:12,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:12,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:56:12,955 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34681,1681527209133' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-15 02:56:12,955 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6f932805[Count = 0] remaining members to acquire global barrier 2023-04-15 02:56:12,955 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-15 02:56:12,955 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:56:12,956 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:56:12,956 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:56:12,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:56:12,956 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-15 02:56:12,956 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-15 02:56:12,956 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34681,1681527209133' in zk 2023-04-15 02:56:12,956 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:12,956 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-15 02:56:12,957 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-15 02:56:12,958 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-15 02:56:12,957 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:12,958 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-15 02:56:12,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:12,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:56:12,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:56:12,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:56:12,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:56:12,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:56:12,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:12,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:56:12,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:56:12,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:12,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34681,1681527209133': 2023-04-15 02:56:12,962 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34681,1681527209133' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-15 02:56:12,962 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-15 02:56:12,962 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-15 02:56:12,962 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-15 02:56:12,962 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-15 02:56:12,962 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-15 02:56:12,972 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:56:12,972 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:56:12,972 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:56:12,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:56:12,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:56:12,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:56:12,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:56:12,972 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=20e761db5975fb9082f870bf056bd078, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:56:12,972 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-15 02:56:12,972 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34681] ipc.CallRunner(144): callId: 121 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:48664 deadline: 1681527432969, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=20e761db5975fb9082f870bf056bd078, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:12,972 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-15 02:56:12,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-15 02:56:12,973 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=20e761db5975fb9082f870bf056bd078, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:56:12,973 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34681] ipc.CallRunner(144): callId: 143 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:48586 deadline: 1681527432970, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=20e761db5975fb9082f870bf056bd078, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:12,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:56:12,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:56:12,973 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:12,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:56:12,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:56:12,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:56:12,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:12,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:56:12,975 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:12,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:56:12,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:12,979 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:12,979 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-15 02:56:12,979 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-15 02:56:12,979 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-15 02:56:12,979 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-15 02:56:12,979 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:56:12,979 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-15 02:56:12,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-15 02:56:12,979 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-15 02:56:12,979 DEBUG [Thread-3439] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-15 02:56:12,979 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:56:12,979 DEBUG [Thread-3439] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-15 02:56:12,979 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:12,980 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:56:12,980 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:56:12,980 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:56:12,980 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-15 02:56:12,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-15 02:56:13,007 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=20e761db5975fb9082f870bf056bd078, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:56:13,007 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34681] ipc.CallRunner(144): callId: 123 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:48624 deadline: 1681527433005, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=20e761db5975fb9082f870bf056bd078, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:13,009 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=20e761db5975fb9082f870bf056bd078, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:56:13,009 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=20e761db5975fb9082f870bf056bd078, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:56:13,009 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34681] ipc.CallRunner(144): callId: 116 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:48648 deadline: 1681527433006, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=20e761db5975fb9082f870bf056bd078, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:13,009 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34681] ipc.CallRunner(144): callId: 115 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:48666 deadline: 1681527433006, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=20e761db5975fb9082f870bf056bd078, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:13,074 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=20e761db5975fb9082f870bf056bd078, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:56:13,074 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34681] ipc.CallRunner(144): callId: 123 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:48664 deadline: 1681527433073, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=20e761db5975fb9082f870bf056bd078, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:13,075 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=20e761db5975fb9082f870bf056bd078, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:56:13,075 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34681] ipc.CallRunner(144): callId: 145 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:48586 deadline: 1681527433074, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=20e761db5975fb9082f870bf056bd078, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:13,080 DEBUG [Thread-3439] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-15 02:56:13,080 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-15 02:56:13,081 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-15 02:56:13,082 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-15 02:56:13,082 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-15 02:56:13,082 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-15 02:56:13,082 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-15 02:56:13,082 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-15 02:56:13,082 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-15 02:56:13,082 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:56:13,082 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:56:13,083 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-15 02:56:13,083 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:13,083 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-15 02:56:13,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-15 02:56:13,083 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:13,083 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-15 02:56:13,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:56:13,083 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:56:13,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-15 02:56:13,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:56:13,084 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-15 02:56:13,084 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-15 02:56:13,084 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-15 02:56:13,084 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-15 02:56:13,084 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-15 02:56:13,084 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-15 02:56:13,084 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-15 02:56:13,084 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-15 02:56:13,084 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681527356526.20e761db5975fb9082f870bf056bd078. 2023-04-15 02:56:13,084 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681527356526.20e761db5975fb9082f870bf056bd078. started... 2023-04-15 02:56:13,084 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681527356526.20e761db5975fb9082f870bf056bd078. as already flushing 2023-04-15 02:56:13,084 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681527356526.20e761db5975fb9082f870bf056bd078. 2023-04-15 02:56:13,084 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-15 02:56:13,084 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-15 02:56:13,085 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:13,085 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-15 02:56:13,085 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34681,1681527209133' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-15 02:56:13,085 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:13,085 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:56:13,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:13,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:56:13,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:56:13,086 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:56:13,086 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-15 02:56:13,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:56:13,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:56:13,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:56:13,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:13,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:56:13,087 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34681,1681527209133' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-15 02:56:13,087 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-15 02:56:13,087 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2db8ba95[Count = 0] remaining members to acquire global barrier 2023-04-15 02:56:13,087 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:56:13,087 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:56:13,087 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:56:13,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:56:13,087 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:13,087 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-15 02:56:13,087 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-15 02:56:13,087 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-15 02:56:13,088 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34681,1681527209133' in zk 2023-04-15 02:56:13,088 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:13,088 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-15 02:56:13,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:13,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:56:13,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:56:13,088 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-15 02:56:13,088 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-15 02:56:13,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:56:13,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:56:13,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:56:13,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:13,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:56:13,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:56:13,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:13,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34681,1681527209133': 2023-04-15 02:56:13,090 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34681,1681527209133' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-15 02:56:13,090 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-15 02:56:13,090 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-15 02:56:13,090 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-15 02:56:13,090 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-15 02:56:13,090 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-15 02:56:13,091 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:56:13,091 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:56:13,091 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:56:13,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:56:13,091 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-15 02:56:13,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:56:13,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:56:13,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:56:13,092 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-15 02:56:13,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-15 02:56:13,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:56:13,092 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:13,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:56:13,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:56:13,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:56:13,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:56:13,093 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:13,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:13,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:56:13,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:56:13,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:13,094 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:13,094 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:56:13,094 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-15 02:56:13,094 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:56:13,094 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-15 02:56:13,094 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-15 02:56:13,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-15 02:56:13,094 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-15 02:56:13,094 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-15 02:56:13,094 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-15 02:56:13,094 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:13,094 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:56:13,094 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:56:13,094 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:56:13,094 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-15 02:56:13,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-15 02:56:13,094 DEBUG [Thread-3439] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-15 02:56:13,094 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-15 02:56:13,094 DEBUG [Thread-3439] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-15 02:56:13,195 DEBUG [Thread-3439] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-15 02:56:13,195 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-15 02:56:13,208 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-15 02:56:13,209 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-15 02:56:13,209 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-15 02:56:13,209 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=20e761db5975fb9082f870bf056bd078, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:56:13,209 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-15 02:56:13,209 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34681] ipc.CallRunner(144): callId: 125 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:48624 deadline: 1681527433208, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=20e761db5975fb9082f870bf056bd078, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:13,210 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-15 02:56:13,210 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-15 02:56:13,210 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-15 02:56:13,210 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:56:13,210 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:56:13,211 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-15 02:56:13,211 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:13,211 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-15 02:56:13,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-15 02:56:13,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:56:13,211 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:56:13,211 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:13,211 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-15 02:56:13,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-15 02:56:13,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:56:13,212 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-15 02:56:13,212 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-15 02:56:13,212 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-15 02:56:13,212 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-15 02:56:13,212 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-15 02:56:13,212 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-15 02:56:13,212 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-15 02:56:13,212 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681527356526.20e761db5975fb9082f870bf056bd078. 2023-04-15 02:56:13,212 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681527356526.20e761db5975fb9082f870bf056bd078. started... 2023-04-15 02:56:13,212 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681527356526.20e761db5975fb9082f870bf056bd078. as already flushing 2023-04-15 02:56:13,212 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681527356526.20e761db5975fb9082f870bf056bd078. 2023-04-15 02:56:13,212 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-15 02:56:13,212 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-15 02:56:13,212 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:13,212 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-15 02:56:13,213 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34681,1681527209133' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-15 02:56:13,213 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=20e761db5975fb9082f870bf056bd078, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:56:13,213 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34681] ipc.CallRunner(144): callId: 118 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:48648 deadline: 1681527433210, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=20e761db5975fb9082f870bf056bd078, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:13,213 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=20e761db5975fb9082f870bf056bd078, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:56:13,213 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34681] ipc.CallRunner(144): callId: 117 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:48666 deadline: 1681527433212, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=20e761db5975fb9082f870bf056bd078, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:13,213 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:56:13,213 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:13,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:13,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:56:13,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:56:13,214 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:56:13,214 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-15 02:56:13,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:56:13,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:56:13,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:56:13,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:13,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:56:13,215 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34681,1681527209133' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-15 02:56:13,215 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@53fbaf13[Count = 0] remaining members to acquire global barrier 2023-04-15 02:56:13,215 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-15 02:56:13,215 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:56:13,216 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:56:13,216 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:56:13,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:56:13,216 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-15 02:56:13,216 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-15 02:56:13,216 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:13,216 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-15 02:56:13,216 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34681,1681527209133' in zk 2023-04-15 02:56:13,217 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:13,217 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-15 02:56:13,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:13,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:56:13,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:56:13,217 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-15 02:56:13,218 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-15 02:56:13,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:56:13,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:56:13,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:56:13,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:13,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:56:13,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:56:13,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:13,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34681,1681527209133': 2023-04-15 02:56:13,219 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34681,1681527209133' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-15 02:56:13,219 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-15 02:56:13,219 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-15 02:56:13,219 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-15 02:56:13,219 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-15 02:56:13,219 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-15 02:56:13,221 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:56:13,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:56:13,221 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:56:13,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:56:13,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:56:13,222 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:56:13,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:56:13,222 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-15 02:56:13,222 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-15 02:56:13,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-15 02:56:13,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:56:13,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:56:13,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:56:13,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:56:13,223 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:13,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:56:13,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:13,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:56:13,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:56:13,224 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:13,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:13,225 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-15 02:56:13,225 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-15 02:56:13,225 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:13,225 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:56:13,225 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-15 02:56:13,225 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-15 02:56:13,225 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-15 02:56:13,225 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-15 02:56:13,226 DEBUG [Thread-3439] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-15 02:56:13,226 DEBUG [Thread-3439] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-15 02:56:13,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-15 02:56:13,225 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:56:13,226 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:13,226 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:56:13,226 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:56:13,226 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:56:13,226 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-15 02:56:13,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-15 02:56:13,276 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=20e761db5975fb9082f870bf056bd078, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:56:13,276 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34681] ipc.CallRunner(144): callId: 125 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:48664 deadline: 1681527433275, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=20e761db5975fb9082f870bf056bd078, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:13,278 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=20e761db5975fb9082f870bf056bd078, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:56:13,278 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34681] ipc.CallRunner(144): callId: 147 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:48586 deadline: 1681527433277, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=20e761db5975fb9082f870bf056bd078, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:13,326 DEBUG [Thread-3439] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-15 02:56:13,326 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-15 02:56:13,327 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-15 02:56:13,332 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-15 02:56:13,332 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-15 02:56:13,332 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-15 02:56:13,332 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-15 02:56:13,332 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-15 02:56:13,332 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-15 02:56:13,332 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:56:13,332 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:56:13,333 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:13,333 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-15 02:56:13,333 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-15 02:56:13,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-15 02:56:13,333 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:13,333 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-15 02:56:13,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:56:13,333 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:56:13,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-15 02:56:13,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:56:13,334 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-15 02:56:13,334 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-15 02:56:13,334 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-15 02:56:13,334 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-15 02:56:13,334 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-15 02:56:13,334 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-15 02:56:13,334 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-15 02:56:13,334 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-15 02:56:13,334 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681527356526.20e761db5975fb9082f870bf056bd078. 2023-04-15 02:56:13,334 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681527356526.20e761db5975fb9082f870bf056bd078. started... 2023-04-15 02:56:13,334 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681527356526.20e761db5975fb9082f870bf056bd078. as already flushing 2023-04-15 02:56:13,335 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681527356526.20e761db5975fb9082f870bf056bd078. 2023-04-15 02:56:13,335 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-15 02:56:13,335 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-15 02:56:13,335 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:13,335 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-15 02:56:13,335 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34681,1681527209133' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-15 02:56:13,335 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:56:13,335 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:13,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:13,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:56:13,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:56:13,335 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:56:13,336 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-15 02:56:13,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:56:13,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:56:13,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:56:13,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:13,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:56:13,337 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34681,1681527209133' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-15 02:56:13,337 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-15 02:56:13,337 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1b917864[Count = 0] remaining members to acquire global barrier 2023-04-15 02:56:13,337 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:56:13,337 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:56:13,337 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:56:13,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:56:13,337 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-15 02:56:13,337 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:13,337 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-15 02:56:13,337 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-15 02:56:13,337 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34681,1681527209133' in zk 2023-04-15 02:56:13,338 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:13,338 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-15 02:56:13,338 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-15 02:56:13,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:13,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:56:13,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:56:13,338 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-15 02:56:13,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:56:13,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:56:13,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:56:13,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:13,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:56:13,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:56:13,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:13,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34681,1681527209133': 2023-04-15 02:56:13,340 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34681,1681527209133' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-15 02:56:13,340 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-15 02:56:13,340 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-15 02:56:13,340 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-15 02:56:13,340 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-15 02:56:13,340 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-15 02:56:13,341 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:56:13,341 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:56:13,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:56:13,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:56:13,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:56:13,341 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:56:13,341 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-15 02:56:13,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:56:13,341 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:13,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:56:13,341 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-15 02:56:13,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-15 02:56:13,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:56:13,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:56:13,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:56:13,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:56:13,342 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:13,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:13,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:56:13,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:56:13,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:13,344 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:13,344 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-15 02:56:13,344 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:56:13,344 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-15 02:56:13,344 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-15 02:56:13,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-15 02:56:13,344 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:56:13,344 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:13,344 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:56:13,344 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:56:13,344 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-15 02:56:13,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-15 02:56:13,344 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:56:13,344 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-15 02:56:13,344 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-15 02:56:13,344 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-15 02:56:13,344 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-15 02:56:13,344 DEBUG [Thread-3439] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-15 02:56:13,344 DEBUG [Thread-3439] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-15 02:56:13,372 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=268 (bloomFilter=true), to=hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/20e761db5975fb9082f870bf056bd078/.tmp/C/bc23e3640fa7498094a8d56ae64ac1a1 2023-04-15 02:56:13,375 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/20e761db5975fb9082f870bf056bd078/.tmp/A/7ff283029d7748108529cd72184d2d13 as hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/20e761db5975fb9082f870bf056bd078/A/7ff283029d7748108529cd72184d2d13 2023-04-15 02:56:13,377 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/20e761db5975fb9082f870bf056bd078/A/7ff283029d7748108529cd72184d2d13, entries=300, sequenceid=268, filesize=46.1 K 2023-04-15 02:56:13,378 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/20e761db5975fb9082f870bf056bd078/.tmp/B/82721bd5833e4862be0e0abd0b57cde5 as hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/20e761db5975fb9082f870bf056bd078/B/82721bd5833e4862be0e0abd0b57cde5 2023-04-15 02:56:13,380 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/20e761db5975fb9082f870bf056bd078/B/82721bd5833e4862be0e0abd0b57cde5, entries=150, sequenceid=268, filesize=11.9 K 2023-04-15 02:56:13,380 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/20e761db5975fb9082f870bf056bd078/.tmp/C/bc23e3640fa7498094a8d56ae64ac1a1 as hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/20e761db5975fb9082f870bf056bd078/C/bc23e3640fa7498094a8d56ae64ac1a1 2023-04-15 02:56:13,383 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/20e761db5975fb9082f870bf056bd078/C/bc23e3640fa7498094a8d56ae64ac1a1, entries=150, sequenceid=268, filesize=11.9 K 2023-04-15 02:56:13,383 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=127.47 KB/130530 for 20e761db5975fb9082f870bf056bd078 in 508ms, sequenceid=268, compaction requested=true 2023-04-15 02:56:13,383 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 20e761db5975fb9082f870bf056bd078: 2023-04-15 02:56:13,383 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-15 02:56:13,383 DEBUG [RS:0;jenkins-hbase20:34681-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-15 02:56:13,383 DEBUG [RS:0;jenkins-hbase20:34681-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-15 02:56:13,383 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-15 02:56:13,384 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-15 02:56:13,384 DEBUG [RS:0;jenkins-hbase20:34681-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 99406 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-15 02:56:13,384 DEBUG [RS:0;jenkins-hbase20:34681-shortCompactions-0] regionserver.HStore(1912): 20e761db5975fb9082f870bf056bd078/A is initiating minor compaction (all files) 2023-04-15 02:56:13,384 DEBUG [RS:0;jenkins-hbase20:34681-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36702 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-15 02:56:13,384 INFO [RS:0;jenkins-hbase20:34681-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 20e761db5975fb9082f870bf056bd078/A in TestAcidGuarantees,,1681527356526.20e761db5975fb9082f870bf056bd078. 2023-04-15 02:56:13,384 DEBUG [RS:0;jenkins-hbase20:34681-longCompactions-0] regionserver.HStore(1912): 20e761db5975fb9082f870bf056bd078/B is initiating minor compaction (all files) 2023-04-15 02:56:13,384 INFO [RS:0;jenkins-hbase20:34681-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 20e761db5975fb9082f870bf056bd078/B in TestAcidGuarantees,,1681527356526.20e761db5975fb9082f870bf056bd078. 2023-04-15 02:56:13,384 INFO [RS:0;jenkins-hbase20:34681-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/20e761db5975fb9082f870bf056bd078/A/a59ed8a7c4bd40a6879c548ffb1870d2, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/20e761db5975fb9082f870bf056bd078/A/8c4dc0ae326943b4ae95dc85a0dbc4e5, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/20e761db5975fb9082f870bf056bd078/A/7ff283029d7748108529cd72184d2d13] into tmpdir=hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/20e761db5975fb9082f870bf056bd078/.tmp, totalSize=97.1 K 2023-04-15 02:56:13,384 INFO [RS:0;jenkins-hbase20:34681-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/20e761db5975fb9082f870bf056bd078/B/a7e7f41bd36a4aefbcf65317d8aa8ab8, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/20e761db5975fb9082f870bf056bd078/B/a48b06a3343d435b86de97231507762d, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/20e761db5975fb9082f870bf056bd078/B/82721bd5833e4862be0e0abd0b57cde5] into tmpdir=hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/20e761db5975fb9082f870bf056bd078/.tmp, totalSize=35.8 K 2023-04-15 02:56:13,385 DEBUG [RS:0;jenkins-hbase20:34681-shortCompactions-0] compactions.Compactor(207): Compacting a59ed8a7c4bd40a6879c548ffb1870d2, keycount=150, bloomtype=ROW, size=25.7 K, encoding=NONE, compression=NONE, seqNum=229, earliestPutTs=1681527369643 2023-04-15 02:56:13,385 DEBUG [RS:0;jenkins-hbase20:34681-longCompactions-0] compactions.Compactor(207): Compacting a7e7f41bd36a4aefbcf65317d8aa8ab8, keycount=150, bloomtype=ROW, size=12.2 K, encoding=NONE, compression=NONE, seqNum=229, earliestPutTs=1681527369643 2023-04-15 02:56:13,385 DEBUG [RS:0;jenkins-hbase20:34681-shortCompactions-0] compactions.Compactor(207): Compacting 8c4dc0ae326943b4ae95dc85a0dbc4e5, keycount=150, bloomtype=ROW, size=25.3 K, encoding=NONE, compression=NONE, seqNum=249, earliestPutTs=1681527370809 2023-04-15 02:56:13,385 DEBUG [RS:0;jenkins-hbase20:34681-longCompactions-0] compactions.Compactor(207): Compacting a48b06a3343d435b86de97231507762d, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=249, earliestPutTs=1681527370809 2023-04-15 02:56:13,385 DEBUG [RS:0;jenkins-hbase20:34681-shortCompactions-0] compactions.Compactor(207): Compacting 7ff283029d7748108529cd72184d2d13, keycount=300, bloomtype=ROW, size=46.1 K, encoding=NONE, compression=NONE, seqNum=268, earliestPutTs=1681527372842 2023-04-15 02:56:13,385 DEBUG [RS:0;jenkins-hbase20:34681-longCompactions-0] compactions.Compactor(207): Compacting 82721bd5833e4862be0e0abd0b57cde5, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=268, earliestPutTs=1681527372872 2023-04-15 02:56:13,390 INFO [RS:0;jenkins-hbase20:34681-longCompactions-0] throttle.PressureAwareThroughputController(145): 20e761db5975fb9082f870bf056bd078#B#compaction#489 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-15 02:56:13,396 INFO [RS:0;jenkins-hbase20:34681-shortCompactions-0] throttle.PressureAwareThroughputController(145): 20e761db5975fb9082f870bf056bd078#A#compaction#488 average throughput is 2.82 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-15 02:56:13,402 DEBUG [RS:0;jenkins-hbase20:34681-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/20e761db5975fb9082f870bf056bd078/.tmp/B/770f8ea7fafd43e1b405486caae1c1bf as hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/20e761db5975fb9082f870bf056bd078/B/770f8ea7fafd43e1b405486caae1c1bf 2023-04-15 02:56:13,406 INFO [RS:0;jenkins-hbase20:34681-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 20e761db5975fb9082f870bf056bd078/B of 20e761db5975fb9082f870bf056bd078 into 770f8ea7fafd43e1b405486caae1c1bf(size=12.4 K), total size for store is 12.4 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-15 02:56:13,406 DEBUG [RS:0;jenkins-hbase20:34681-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 20e761db5975fb9082f870bf056bd078: 2023-04-15 02:56:13,406 INFO [RS:0;jenkins-hbase20:34681-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681527356526.20e761db5975fb9082f870bf056bd078., storeName=20e761db5975fb9082f870bf056bd078/B, priority=13, startTime=1681527373383; duration=0sec 2023-04-15 02:56:13,406 DEBUG [RS:0;jenkins-hbase20:34681-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-15 02:56:13,407 DEBUG [RS:0;jenkins-hbase20:34681-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 2 store files, 0 compacting, 2 eligible, 16 blocking 2023-04-15 02:56:13,407 DEBUG [RS:0;jenkins-hbase20:34681-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 0 files of size 0 starting at candidate #-1 after considering 0 permutations with 0 in ratio 2023-04-15 02:56:13,407 DEBUG [RS:0;jenkins-hbase20:34681-longCompactions-0] compactions.SortedCompactionPolicy(232): Not compacting files because we only have 0 files ready for compaction. Need 3 to initiate. 2023-04-15 02:56:13,407 DEBUG [RS:0;jenkins-hbase20:34681-longCompactions-0] regionserver.CompactSplit(415): Not compacting TestAcidGuarantees,,1681527356526.20e761db5975fb9082f870bf056bd078. because compaction request was cancelled 2023-04-15 02:56:13,445 DEBUG [Thread-3439] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-15 02:56:13,445 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-15 02:56:13,446 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-15 02:56:13,447 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-15 02:56:13,447 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-15 02:56:13,447 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-15 02:56:13,447 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-15 02:56:13,447 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-15 02:56:13,447 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-15 02:56:13,448 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:56:13,448 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:56:13,448 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:13,448 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-15 02:56:13,448 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-15 02:56:13,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-15 02:56:13,448 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:13,448 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-15 02:56:13,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:56:13,449 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:56:13,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-15 02:56:13,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:56:13,449 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-15 02:56:13,449 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-15 02:56:13,449 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-15 02:56:13,449 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-15 02:56:13,449 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-15 02:56:13,449 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-15 02:56:13,450 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-15 02:56:13,450 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-15 02:56:13,450 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681527356526.20e761db5975fb9082f870bf056bd078. 2023-04-15 02:56:13,450 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681527356526.20e761db5975fb9082f870bf056bd078. started... 2023-04-15 02:56:13,450 INFO [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 20e761db5975fb9082f870bf056bd078 3/3 column families, dataSize=127.47 KB heapSize=365.06 KB 2023-04-15 02:56:13,450 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 20e761db5975fb9082f870bf056bd078, store=A 2023-04-15 02:56:13,450 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-15 02:56:13,450 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 20e761db5975fb9082f870bf056bd078, store=B 2023-04-15 02:56:13,450 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-15 02:56:13,450 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 20e761db5975fb9082f870bf056bd078, store=C 2023-04-15 02:56:13,450 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-15 02:56:13,514 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34681] regionserver.HRegion(9158): Flush requested on 20e761db5975fb9082f870bf056bd078 2023-04-15 02:56:13,514 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681527356526.20e761db5975fb9082f870bf056bd078. as already flushing 2023-04-15 02:56:13,523 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=20e761db5975fb9082f870bf056bd078, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:56:13,523 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34681] ipc.CallRunner(144): callId: 121 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:48666 deadline: 1681527433521, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=20e761db5975fb9082f870bf056bd078, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:13,524 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=20e761db5975fb9082f870bf056bd078, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:56:13,524 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34681] ipc.CallRunner(144): callId: 122 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:48648 deadline: 1681527433522, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=20e761db5975fb9082f870bf056bd078, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:13,524 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=20e761db5975fb9082f870bf056bd078, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:56:13,524 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34681] ipc.CallRunner(144): callId: 131 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:48624 deadline: 1681527433522, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=20e761db5975fb9082f870bf056bd078, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:13,580 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=20e761db5975fb9082f870bf056bd078, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:56:13,581 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34681] ipc.CallRunner(144): callId: 127 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:48664 deadline: 1681527433579, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=20e761db5975fb9082f870bf056bd078, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:13,584 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=20e761db5975fb9082f870bf056bd078, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:56:13,584 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34681] ipc.CallRunner(144): callId: 149 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:48586 deadline: 1681527433581, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=20e761db5975fb9082f870bf056bd078, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:13,626 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=20e761db5975fb9082f870bf056bd078, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:56:13,626 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34681] ipc.CallRunner(144): callId: 123 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:48666 deadline: 1681527433624, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=20e761db5975fb9082f870bf056bd078, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:13,626 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=20e761db5975fb9082f870bf056bd078, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:56:13,626 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34681] ipc.CallRunner(144): callId: 124 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:48648 deadline: 1681527433624, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=20e761db5975fb9082f870bf056bd078, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:13,626 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=20e761db5975fb9082f870bf056bd078, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:56:13,627 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34681] ipc.CallRunner(144): callId: 133 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:48624 deadline: 1681527433625, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=20e761db5975fb9082f870bf056bd078, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:13,810 DEBUG [RS:0;jenkins-hbase20:34681-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/20e761db5975fb9082f870bf056bd078/.tmp/A/3825956b66294ff9b1219e17318fe437 as hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/20e761db5975fb9082f870bf056bd078/A/3825956b66294ff9b1219e17318fe437 2023-04-15 02:56:13,812 INFO [RS:0;jenkins-hbase20:34681-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 20e761db5975fb9082f870bf056bd078/A of 20e761db5975fb9082f870bf056bd078 into 3825956b66294ff9b1219e17318fe437(size=26.0 K), total size for store is 26.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-15 02:56:13,812 DEBUG [RS:0;jenkins-hbase20:34681-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 20e761db5975fb9082f870bf056bd078: 2023-04-15 02:56:13,812 INFO [RS:0;jenkins-hbase20:34681-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681527356526.20e761db5975fb9082f870bf056bd078., storeName=20e761db5975fb9082f870bf056bd078/A, priority=13, startTime=1681527373383; duration=0sec 2023-04-15 02:56:13,813 DEBUG [RS:0;jenkins-hbase20:34681-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-15 02:56:13,828 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=20e761db5975fb9082f870bf056bd078, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:56:13,828 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34681] ipc.CallRunner(144): callId: 125 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:48666 deadline: 1681527433827, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=20e761db5975fb9082f870bf056bd078, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:13,829 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=20e761db5975fb9082f870bf056bd078, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:56:13,829 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34681] ipc.CallRunner(144): callId: 126 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:48648 deadline: 1681527433827, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=20e761db5975fb9082f870bf056bd078, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:13,830 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=20e761db5975fb9082f870bf056bd078, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:56:13,830 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34681] ipc.CallRunner(144): callId: 135 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:48624 deadline: 1681527433828, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=20e761db5975fb9082f870bf056bd078, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:13,871 INFO [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304157b3b2b4285f7431497122d84df30463c to hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304157b3b2b4285f7431497122d84df30463c 2023-04-15 02:56:13,874 INFO [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=305, memsize=42.5 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/20e761db5975fb9082f870bf056bd078/.tmp/A/f7be98643a9b4774805a2ec65b40aa86 2023-04-15 02:56:14,087 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=20e761db5975fb9082f870bf056bd078, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:56:14,087 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34681] ipc.CallRunner(144): callId: 129 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:48664 deadline: 1681527434085, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=20e761db5975fb9082f870bf056bd078, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:14,087 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=20e761db5975fb9082f870bf056bd078, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:56:14,087 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34681] ipc.CallRunner(144): callId: 151 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:48586 deadline: 1681527434086, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=20e761db5975fb9082f870bf056bd078, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:14,130 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=20e761db5975fb9082f870bf056bd078, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:56:14,130 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34681] ipc.CallRunner(144): callId: 127 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:48666 deadline: 1681527434129, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=20e761db5975fb9082f870bf056bd078, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:14,131 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=20e761db5975fb9082f870bf056bd078, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:56:14,131 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34681] ipc.CallRunner(144): callId: 128 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:48648 deadline: 1681527434129, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=20e761db5975fb9082f870bf056bd078, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:14,131 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=20e761db5975fb9082f870bf056bd078, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:56:14,131 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34681] ipc.CallRunner(144): callId: 137 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:48624 deadline: 1681527434131, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=20e761db5975fb9082f870bf056bd078, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:14,282 INFO [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=305 (bloomFilter=true), to=hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/20e761db5975fb9082f870bf056bd078/.tmp/B/706ef098332d4a7a9cafb947d28e285d 2023-04-15 02:56:14,634 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=20e761db5975fb9082f870bf056bd078, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:56:14,635 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34681] ipc.CallRunner(144): callId: 130 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:48648 deadline: 1681527434633, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=20e761db5975fb9082f870bf056bd078, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:14,635 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=20e761db5975fb9082f870bf056bd078, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:56:14,635 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34681] ipc.CallRunner(144): callId: 139 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:48624 deadline: 1681527434633, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=20e761db5975fb9082f870bf056bd078, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:14,635 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=20e761db5975fb9082f870bf056bd078, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:56:14,635 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34681] ipc.CallRunner(144): callId: 129 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:48666 deadline: 1681527434635, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=20e761db5975fb9082f870bf056bd078, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:14,689 INFO [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=305 (bloomFilter=true), to=hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/20e761db5975fb9082f870bf056bd078/.tmp/C/8828de42aa92431d8eddbf66abcc5537 2023-04-15 02:56:14,692 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/20e761db5975fb9082f870bf056bd078/.tmp/A/f7be98643a9b4774805a2ec65b40aa86 as hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/20e761db5975fb9082f870bf056bd078/A/f7be98643a9b4774805a2ec65b40aa86 2023-04-15 02:56:14,694 INFO [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/20e761db5975fb9082f870bf056bd078/A/f7be98643a9b4774805a2ec65b40aa86, entries=150, sequenceid=305, filesize=25.4 K 2023-04-15 02:56:14,695 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/20e761db5975fb9082f870bf056bd078/.tmp/B/706ef098332d4a7a9cafb947d28e285d as hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/20e761db5975fb9082f870bf056bd078/B/706ef098332d4a7a9cafb947d28e285d 2023-04-15 02:56:14,697 INFO [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/20e761db5975fb9082f870bf056bd078/B/706ef098332d4a7a9cafb947d28e285d, entries=150, sequenceid=305, filesize=11.9 K 2023-04-15 02:56:14,698 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/20e761db5975fb9082f870bf056bd078/.tmp/C/8828de42aa92431d8eddbf66abcc5537 as hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/20e761db5975fb9082f870bf056bd078/C/8828de42aa92431d8eddbf66abcc5537 2023-04-15 02:56:14,701 INFO [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/20e761db5975fb9082f870bf056bd078/C/8828de42aa92431d8eddbf66abcc5537, entries=150, sequenceid=305, filesize=11.9 K 2023-04-15 02:56:14,701 INFO [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~127.47 KB/130530, heapSize ~365.02 KB/373776, currentSize=53.67 KB/54960 for 20e761db5975fb9082f870bf056bd078 in 1251ms, sequenceid=305, compaction requested=true 2023-04-15 02:56:14,704 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 20e761db5975fb9082f870bf056bd078: 2023-04-15 02:56:14,704 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681527356526.20e761db5975fb9082f870bf056bd078. 2023-04-15 02:56:14,704 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-15 02:56:14,704 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-15 02:56:14,704 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:14,704 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-15 02:56:14,704 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34681,1681527209133' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-15 02:56:14,705 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:14,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:14,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:56:14,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:56:14,705 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:56:14,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:56:14,705 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:56:14,705 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-15 02:56:14,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:56:14,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:56:14,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:14,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:56:14,706 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34681,1681527209133' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-15 02:56:14,706 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6db4b813[Count = 0] remaining members to acquire global barrier 2023-04-15 02:56:14,706 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-15 02:56:14,706 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:56:14,707 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:56:14,707 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:56:14,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:56:14,707 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-15 02:56:14,707 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-15 02:56:14,707 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34681,1681527209133' in zk 2023-04-15 02:56:14,707 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-15 02:56:14,707 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:14,707 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-15 02:56:14,708 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-15 02:56:14,708 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34681,1681527209133' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-15 02:56:14,708 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-15 02:56:14,708 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-15 02:56:14,708 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-15 02:56:14,708 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-15 02:56:14,708 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-15 02:56:14,708 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-15 02:56:14,709 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:56:14,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:56:14,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:56:14,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:56:14,709 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:56:14,709 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-15 02:56:14,709 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:56:14,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:56:14,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:56:14,709 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-15 02:56:14,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-15 02:56:14,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:56:14,709 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:14,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:56:14,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:56:14,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:56:14,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:14,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:56:14,711 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:14,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:56:14,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:14,712 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:14,712 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:56:14,712 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-15 02:56:14,712 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:56:14,712 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-15 02:56:14,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-15 02:56:14,712 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-15 02:56:14,713 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-15 02:56:14,712 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:14,712 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-15 02:56:14,712 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-15 02:56:14,713 DEBUG [Thread-3439] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-15 02:56:14,713 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:56:14,713 DEBUG [Thread-3439] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-15 02:56:14,713 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-15 02:56:14,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-15 02:56:14,713 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:56:14,713 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:56:14,813 DEBUG [Thread-3439] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-15 02:56:14,813 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-15 02:56:14,814 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-15 02:56:14,815 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-15 02:56:14,815 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-15 02:56:14,815 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-15 02:56:14,815 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-15 02:56:14,815 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-15 02:56:14,815 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-15 02:56:14,816 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:56:14,816 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:56:14,816 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-15 02:56:14,816 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-15 02:56:14,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-15 02:56:14,816 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:14,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:56:14,817 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:14,817 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-15 02:56:14,817 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:56:14,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-15 02:56:14,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:56:14,817 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-15 02:56:14,817 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-15 02:56:14,817 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-15 02:56:14,817 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-15 02:56:14,817 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-15 02:56:14,817 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-15 02:56:14,817 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-15 02:56:14,818 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-15 02:56:14,818 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681527356526.20e761db5975fb9082f870bf056bd078. 2023-04-15 02:56:14,818 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681527356526.20e761db5975fb9082f870bf056bd078. started... 2023-04-15 02:56:14,818 INFO [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 20e761db5975fb9082f870bf056bd078 3/3 column families, dataSize=53.67 KB heapSize=160.13 KB 2023-04-15 02:56:14,818 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 20e761db5975fb9082f870bf056bd078, store=A 2023-04-15 02:56:14,818 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-15 02:56:14,818 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 20e761db5975fb9082f870bf056bd078, store=B 2023-04-15 02:56:14,818 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-15 02:56:14,818 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 20e761db5975fb9082f870bf056bd078, store=C 2023-04-15 02:56:14,818 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-15 02:56:14,827 INFO [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230415d125e115f94e44489764a75d1d59518d to hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230415d125e115f94e44489764a75d1d59518d 2023-04-15 02:56:14,831 INFO [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=317, memsize=17.9 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/20e761db5975fb9082f870bf056bd078/.tmp/A/1572e12eb1ac4642b76fca1232b4fd83 2023-04-15 02:56:14,842 INFO [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=317 (bloomFilter=true), to=hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/20e761db5975fb9082f870bf056bd078/.tmp/B/6c7eee15ebd144d8b8b8d11efbf65068 2023-04-15 02:56:15,095 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34681] regionserver.HRegion(9158): Flush requested on 20e761db5975fb9082f870bf056bd078 2023-04-15 02:56:15,095 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681527356526.20e761db5975fb9082f870bf056bd078. as already flushing 2023-04-15 02:56:15,117 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681527375104, max timestamp=1681527375116Num uniques 859; 2023-04-15 02:56:15,117 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-7] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-15 02:56:15,117 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681527375104, max timestamp=1681527375116Num uniques 859; 2023-04-15 02:56:15,117 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-15 02:56:15,118 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681527375104, max timestamp=1681527375116Num uniques 859; 2023-04-15 02:56:15,118 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-3] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-15 02:56:15,144 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=20e761db5975fb9082f870bf056bd078, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:56:15,144 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34681] ipc.CallRunner(144): callId: 169 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:48586 deadline: 1681527435140, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=20e761db5975fb9082f870bf056bd078, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:15,146 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=20e761db5975fb9082f870bf056bd078, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:56:15,146 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34681] ipc.CallRunner(144): callId: 148 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:48664 deadline: 1681527435143, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=20e761db5975fb9082f870bf056bd078, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:15,246 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=20e761db5975fb9082f870bf056bd078, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:56:15,246 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34681] ipc.CallRunner(144): callId: 171 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:48586 deadline: 1681527435245, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=20e761db5975fb9082f870bf056bd078, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:15,248 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=20e761db5975fb9082f870bf056bd078, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:56:15,249 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34681] ipc.CallRunner(144): callId: 150 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:48664 deadline: 1681527435246, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=20e761db5975fb9082f870bf056bd078, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:15,249 INFO [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=317 (bloomFilter=true), to=hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/20e761db5975fb9082f870bf056bd078/.tmp/C/3db79db9269a42dfaf6d4d15d1e01dea 2023-04-15 02:56:15,252 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/20e761db5975fb9082f870bf056bd078/.tmp/A/1572e12eb1ac4642b76fca1232b4fd83 as hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/20e761db5975fb9082f870bf056bd078/A/1572e12eb1ac4642b76fca1232b4fd83 2023-04-15 02:56:15,254 INFO [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/20e761db5975fb9082f870bf056bd078/A/1572e12eb1ac4642b76fca1232b4fd83, entries=150, sequenceid=317, filesize=25.4 K 2023-04-15 02:56:15,255 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/20e761db5975fb9082f870bf056bd078/.tmp/B/6c7eee15ebd144d8b8b8d11efbf65068 as hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/20e761db5975fb9082f870bf056bd078/B/6c7eee15ebd144d8b8b8d11efbf65068 2023-04-15 02:56:15,258 INFO [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/20e761db5975fb9082f870bf056bd078/B/6c7eee15ebd144d8b8b8d11efbf65068, entries=150, sequenceid=317, filesize=11.9 K 2023-04-15 02:56:15,258 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/20e761db5975fb9082f870bf056bd078/.tmp/C/3db79db9269a42dfaf6d4d15d1e01dea as hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/20e761db5975fb9082f870bf056bd078/C/3db79db9269a42dfaf6d4d15d1e01dea 2023-04-15 02:56:15,260 INFO [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/20e761db5975fb9082f870bf056bd078/C/3db79db9269a42dfaf6d4d15d1e01dea, entries=150, sequenceid=317, filesize=11.9 K 2023-04-15 02:56:15,261 INFO [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=127.47 KB/130530 for 20e761db5975fb9082f870bf056bd078 in 443ms, sequenceid=317, compaction requested=true 2023-04-15 02:56:15,261 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 20e761db5975fb9082f870bf056bd078: 2023-04-15 02:56:15,261 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681527356526.20e761db5975fb9082f870bf056bd078. 2023-04-15 02:56:15,261 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-15 02:56:15,261 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-15 02:56:15,261 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:15,261 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-15 02:56:15,261 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34681,1681527209133' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-15 02:56:15,262 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:15,262 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:56:15,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:15,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:56:15,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:56:15,262 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:56:15,262 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-15 02:56:15,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:56:15,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:56:15,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:56:15,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:15,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:56:15,263 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34681,1681527209133' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-15 02:56:15,263 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-15 02:56:15,263 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5b0cb22[Count = 0] remaining members to acquire global barrier 2023-04-15 02:56:15,263 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:56:15,264 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:56:15,264 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:56:15,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:56:15,264 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-15 02:56:15,264 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-15 02:56:15,264 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:15,264 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-15 02:56:15,264 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34681,1681527209133' in zk 2023-04-15 02:56:15,265 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:15,265 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-15 02:56:15,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:15,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:56:15,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:56:15,265 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-15 02:56:15,265 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-15 02:56:15,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:56:15,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:56:15,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:56:15,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:15,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:56:15,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:56:15,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:15,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34681,1681527209133': 2023-04-15 02:56:15,266 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34681,1681527209133' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-15 02:56:15,266 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-15 02:56:15,267 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-15 02:56:15,267 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-15 02:56:15,267 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-15 02:56:15,267 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-15 02:56:15,267 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:56:15,267 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:56:15,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:56:15,267 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:56:15,267 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-15 02:56:15,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:56:15,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:56:15,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:56:15,268 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:15,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:56:15,268 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-15 02:56:15,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-15 02:56:15,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:56:15,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:56:15,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:56:15,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:56:15,269 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:15,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:15,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:56:15,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:56:15,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:15,270 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:15,270 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:56:15,270 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:56:15,270 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:15,270 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-15 02:56:15,270 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-15 02:56:15,270 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-15 02:56:15,270 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-15 02:56:15,270 DEBUG [Thread-3439] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-15 02:56:15,270 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:56:15,270 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-15 02:56:15,270 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:56:15,270 DEBUG [Thread-3439] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-15 02:56:15,270 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-15 02:56:15,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-15 02:56:15,270 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-15 02:56:15,270 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:56:15,271 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-15 02:56:15,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-15 02:56:15,370 DEBUG [Thread-3439] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-15 02:56:15,371 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-15 02:56:15,372 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-15 02:56:15,373 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-15 02:56:15,373 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-15 02:56:15,373 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-15 02:56:15,373 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-15 02:56:15,373 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-15 02:56:15,373 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-15 02:56:15,374 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:56:15,374 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:56:15,374 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-15 02:56:15,374 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:15,374 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-15 02:56:15,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-15 02:56:15,375 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:15,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:56:15,375 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-15 02:56:15,375 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:56:15,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-15 02:56:15,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:56:15,375 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-15 02:56:15,375 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-15 02:56:15,375 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-15 02:56:15,375 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-15 02:56:15,375 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-15 02:56:15,376 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-15 02:56:15,376 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-15 02:56:15,376 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-15 02:56:15,376 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681527356526.20e761db5975fb9082f870bf056bd078. 2023-04-15 02:56:15,376 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681527356526.20e761db5975fb9082f870bf056bd078. started... 2023-04-15 02:56:15,376 INFO [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 20e761db5975fb9082f870bf056bd078 3/3 column families, dataSize=127.47 KB heapSize=365.06 KB 2023-04-15 02:56:15,376 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 20e761db5975fb9082f870bf056bd078, store=A 2023-04-15 02:56:15,376 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-15 02:56:15,376 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 20e761db5975fb9082f870bf056bd078, store=B 2023-04-15 02:56:15,376 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-15 02:56:15,376 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 20e761db5975fb9082f870bf056bd078, store=C 2023-04-15 02:56:15,376 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-15 02:56:15,448 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34681] regionserver.HRegion(9158): Flush requested on 20e761db5975fb9082f870bf056bd078 2023-04-15 02:56:15,449 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681527356526.20e761db5975fb9082f870bf056bd078. as already flushing 2023-04-15 02:56:15,462 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=20e761db5975fb9082f870bf056bd078, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:56:15,462 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34681] ipc.CallRunner(144): callId: 155 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:48664 deadline: 1681527435459, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=20e761db5975fb9082f870bf056bd078, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:15,464 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=20e761db5975fb9082f870bf056bd078, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:56:15,464 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34681] ipc.CallRunner(144): callId: 178 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:48586 deadline: 1681527435462, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=20e761db5975fb9082f870bf056bd078, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:15,564 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=20e761db5975fb9082f870bf056bd078, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:56:15,564 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34681] ipc.CallRunner(144): callId: 157 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:48664 deadline: 1681527435563, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=20e761db5975fb9082f870bf056bd078, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:15,567 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=20e761db5975fb9082f870bf056bd078, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:56:15,567 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34681] ipc.CallRunner(144): callId: 180 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:48586 deadline: 1681527435565, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=20e761db5975fb9082f870bf056bd078, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:15,643 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=20e761db5975fb9082f870bf056bd078, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:56:15,643 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34681] ipc.CallRunner(144): callId: 132 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:48648 deadline: 1681527435641, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=20e761db5975fb9082f870bf056bd078, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:15,643 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=20e761db5975fb9082f870bf056bd078, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:56:15,643 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34681] ipc.CallRunner(144): callId: 131 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:48666 deadline: 1681527435642, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=20e761db5975fb9082f870bf056bd078, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:15,643 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=20e761db5975fb9082f870bf056bd078, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:56:15,644 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34681] ipc.CallRunner(144): callId: 141 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:48624 deadline: 1681527435642, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=20e761db5975fb9082f870bf056bd078, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:15,767 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=20e761db5975fb9082f870bf056bd078, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:56:15,767 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34681] ipc.CallRunner(144): callId: 159 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:48664 deadline: 1681527435766, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=20e761db5975fb9082f870bf056bd078, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:15,769 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=20e761db5975fb9082f870bf056bd078, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:56:15,769 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34681] ipc.CallRunner(144): callId: 182 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:48586 deadline: 1681527435768, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=20e761db5975fb9082f870bf056bd078, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:15,790 INFO [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230415b77ef5b7fa79425abda9841fea27d12c to hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230415b77ef5b7fa79425abda9841fea27d12c 2023-04-15 02:56:16,071 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=20e761db5975fb9082f870bf056bd078, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:56:16,071 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34681] ipc.CallRunner(144): callId: 161 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:48664 deadline: 1681527436069, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=20e761db5975fb9082f870bf056bd078, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:16,073 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=20e761db5975fb9082f870bf056bd078, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:56:16,073 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34681] ipc.CallRunner(144): callId: 184 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:48586 deadline: 1681527436072, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=20e761db5975fb9082f870bf056bd078, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:16,194 INFO [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=353, memsize=42.5 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/20e761db5975fb9082f870bf056bd078/.tmp/A/17188db8659b466dbd46fce2f89d2b35 2023-04-15 02:56:16,577 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=20e761db5975fb9082f870bf056bd078, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:56:16,577 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34681] ipc.CallRunner(144): callId: 163 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:48664 deadline: 1681527436576, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=20e761db5975fb9082f870bf056bd078, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:16,580 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=20e761db5975fb9082f870bf056bd078, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:56:16,580 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34681] ipc.CallRunner(144): callId: 186 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:48586 deadline: 1681527436578, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=20e761db5975fb9082f870bf056bd078, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:16,601 INFO [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=353 (bloomFilter=true), to=hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/20e761db5975fb9082f870bf056bd078/.tmp/B/6abe126288384f4d92aac998836c31e0 2023-04-15 02:56:17,015 INFO [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=353 (bloomFilter=true), to=hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/20e761db5975fb9082f870bf056bd078/.tmp/C/5a7f92071f5645c08e36bdeaaf9bcf47 2023-04-15 02:56:17,019 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/20e761db5975fb9082f870bf056bd078/.tmp/A/17188db8659b466dbd46fce2f89d2b35 as hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/20e761db5975fb9082f870bf056bd078/A/17188db8659b466dbd46fce2f89d2b35 2023-04-15 02:56:17,021 INFO [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/20e761db5975fb9082f870bf056bd078/A/17188db8659b466dbd46fce2f89d2b35, entries=150, sequenceid=353, filesize=25.4 K 2023-04-15 02:56:17,022 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/20e761db5975fb9082f870bf056bd078/.tmp/B/6abe126288384f4d92aac998836c31e0 as hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/20e761db5975fb9082f870bf056bd078/B/6abe126288384f4d92aac998836c31e0 2023-04-15 02:56:17,026 INFO [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/20e761db5975fb9082f870bf056bd078/B/6abe126288384f4d92aac998836c31e0, entries=150, sequenceid=353, filesize=11.9 K 2023-04-15 02:56:17,026 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/20e761db5975fb9082f870bf056bd078/.tmp/C/5a7f92071f5645c08e36bdeaaf9bcf47 as hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/20e761db5975fb9082f870bf056bd078/C/5a7f92071f5645c08e36bdeaaf9bcf47 2023-04-15 02:56:17,029 INFO [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/20e761db5975fb9082f870bf056bd078/C/5a7f92071f5645c08e36bdeaaf9bcf47, entries=150, sequenceid=353, filesize=11.9 K 2023-04-15 02:56:17,030 INFO [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~127.47 KB/130530, heapSize ~365.02 KB/373776, currentSize=53.67 KB/54960 for 20e761db5975fb9082f870bf056bd078 in 1654ms, sequenceid=353, compaction requested=true 2023-04-15 02:56:17,030 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 20e761db5975fb9082f870bf056bd078: 2023-04-15 02:56:17,030 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681527356526.20e761db5975fb9082f870bf056bd078. 2023-04-15 02:56:17,030 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-15 02:56:17,030 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-15 02:56:17,030 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:17,030 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-15 02:56:17,030 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34681,1681527209133' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-15 02:56:17,031 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:17,031 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:56:17,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:17,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:56:17,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:56:17,031 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:56:17,031 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-15 02:56:17,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:56:17,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:56:17,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:56:17,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:17,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:56:17,033 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34681,1681527209133' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-15 02:56:17,033 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@bbd7fb0[Count = 0] remaining members to acquire global barrier 2023-04-15 02:56:17,033 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-15 02:56:17,033 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:56:17,033 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:56:17,033 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:56:17,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:56:17,033 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-15 02:56:17,033 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-15 02:56:17,033 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34681,1681527209133' in zk 2023-04-15 02:56:17,034 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:17,034 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-15 02:56:17,035 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-15 02:56:17,035 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-15 02:56:17,035 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-15 02:56:17,037 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:17,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:17,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:56:17,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:56:17,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:56:17,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:56:17,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:56:17,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:17,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:56:17,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:56:17,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:17,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34681,1681527209133': 2023-04-15 02:56:17,039 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34681,1681527209133' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-15 02:56:17,039 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-15 02:56:17,039 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-15 02:56:17,039 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-15 02:56:17,039 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-15 02:56:17,039 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-15 02:56:17,047 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:56:17,047 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:56:17,047 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:56:17,047 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-15 02:56:17,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:56:17,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:56:17,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:56:17,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:56:17,047 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-15 02:56:17,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-15 02:56:17,047 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:17,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:56:17,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:56:17,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:56:17,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:56:17,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:56:17,048 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:17,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:17,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:56:17,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:56:17,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:17,053 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:17,054 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-15 02:56:17,054 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:56:17,054 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-15 02:56:17,054 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:56:17,054 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-15 02:56:17,054 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-15 02:56:17,054 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-15 02:56:17,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-15 02:56:17,054 DEBUG [Thread-3439] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-15 02:56:17,054 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-15 02:56:17,054 DEBUG [Thread-3439] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-15 02:56:17,054 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:17,054 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:56:17,054 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:56:17,054 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-15 02:56:17,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-15 02:56:17,054 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:56:17,154 DEBUG [Thread-3439] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-15 02:56:17,155 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-15 02:56:17,156 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-15 02:56:17,157 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-15 02:56:17,157 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-15 02:56:17,157 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-15 02:56:17,157 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-15 02:56:17,157 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-15 02:56:17,157 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-15 02:56:17,157 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:56:17,157 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:56:17,158 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:17,158 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-15 02:56:17,158 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-15 02:56:17,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-15 02:56:17,158 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:17,158 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-15 02:56:17,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:56:17,159 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:56:17,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-15 02:56:17,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:56:17,159 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-15 02:56:17,159 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-15 02:56:17,159 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-15 02:56:17,159 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-15 02:56:17,159 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-15 02:56:17,159 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-15 02:56:17,159 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-15 02:56:17,159 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-15 02:56:17,159 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681527356526.20e761db5975fb9082f870bf056bd078. 2023-04-15 02:56:17,159 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681527356526.20e761db5975fb9082f870bf056bd078. started... 2023-04-15 02:56:17,160 INFO [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 20e761db5975fb9082f870bf056bd078 3/3 column families, dataSize=53.67 KB heapSize=160.13 KB 2023-04-15 02:56:17,160 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 20e761db5975fb9082f870bf056bd078, store=A 2023-04-15 02:56:17,160 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-15 02:56:17,160 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 20e761db5975fb9082f870bf056bd078, store=B 2023-04-15 02:56:17,160 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-15 02:56:17,160 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 20e761db5975fb9082f870bf056bd078, store=C 2023-04-15 02:56:17,160 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-15 02:56:17,569 INFO [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023041570800ab6e65541ff9ec74ec1decb3627 to hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023041570800ab6e65541ff9ec74ec1decb3627 2023-04-15 02:56:17,585 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34681] regionserver.HRegion(9158): Flush requested on 20e761db5975fb9082f870bf056bd078 2023-04-15 02:56:17,585 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681527356526.20e761db5975fb9082f870bf056bd078. as already flushing 2023-04-15 02:56:17,615 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1681527377598, max timestamp=1681527377614Num uniques 859; 2023-04-15 02:56:17,615 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1681527377598, max timestamp=1681527377614Num uniques 859; 2023-04-15 02:56:17,615 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-5] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-15 02:56:17,615 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-9] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-15 02:56:17,616 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1681527377598, max timestamp=1681527377614Num uniques 859; 2023-04-15 02:56:17,616 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-6] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-15 02:56:17,646 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=20e761db5975fb9082f870bf056bd078, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:56:17,646 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=20e761db5975fb9082f870bf056bd078, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:56:17,646 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34681] ipc.CallRunner(144): callId: 203 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:48586 deadline: 1681527437643, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=20e761db5975fb9082f870bf056bd078, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:17,646 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34681] ipc.CallRunner(144): callId: 182 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:48664 deadline: 1681527437644, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=20e761db5975fb9082f870bf056bd078, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:17,652 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=20e761db5975fb9082f870bf056bd078, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:56:17,652 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34681] ipc.CallRunner(144): callId: 134 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:48648 deadline: 1681527437650, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=20e761db5975fb9082f870bf056bd078, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:17,653 DEBUG [Thread-3437] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4130 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=20e761db5975fb9082f870bf056bd078, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_1' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1681527356526.20e761db5975fb9082f870bf056bd078., hostname=jenkins-hbase20.apache.org,34681,1681527209133, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=20e761db5975fb9082f870bf056bd078, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor31.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=20e761db5975fb9082f870bf056bd078, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-15 02:56:17,655 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=20e761db5975fb9082f870bf056bd078, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:56:17,656 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34681] ipc.CallRunner(144): callId: 143 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:48624 deadline: 1681527437653, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=20e761db5975fb9082f870bf056bd078, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:17,657 DEBUG [Thread-3435] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4134 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=20e761db5975fb9082f870bf056bd078, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_0' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1681527356526.20e761db5975fb9082f870bf056bd078., hostname=jenkins-hbase20.apache.org,34681,1681527209133, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=20e761db5975fb9082f870bf056bd078, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor31.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=20e761db5975fb9082f870bf056bd078, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-15 02:56:17,658 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=20e761db5975fb9082f870bf056bd078, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:56:17,659 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34681] ipc.CallRunner(144): callId: 133 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:48666 deadline: 1681527437654, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=20e761db5975fb9082f870bf056bd078, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:17,660 DEBUG [Thread-3433] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4138 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=20e761db5975fb9082f870bf056bd078, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_1' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1681527356526.20e761db5975fb9082f870bf056bd078., hostname=jenkins-hbase20.apache.org,34681,1681527209133, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=20e761db5975fb9082f870bf056bd078, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor31.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=20e761db5975fb9082f870bf056bd078, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-15 02:56:17,748 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=20e761db5975fb9082f870bf056bd078, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:56:17,748 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34681] ipc.CallRunner(144): callId: 184 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:48664 deadline: 1681527437747, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=20e761db5975fb9082f870bf056bd078, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:17,748 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=20e761db5975fb9082f870bf056bd078, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:56:17,749 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34681] ipc.CallRunner(144): callId: 205 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:48586 deadline: 1681527437747, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=20e761db5975fb9082f870bf056bd078, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:17,952 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=20e761db5975fb9082f870bf056bd078, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:56:17,952 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34681] ipc.CallRunner(144): callId: 186 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:48664 deadline: 1681527437950, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=20e761db5975fb9082f870bf056bd078, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:17,952 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=20e761db5975fb9082f870bf056bd078, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:56:17,952 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34681] ipc.CallRunner(144): callId: 207 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:48586 deadline: 1681527437950, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=20e761db5975fb9082f870bf056bd078, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:17,973 INFO [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=364, memsize=17.9 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/20e761db5975fb9082f870bf056bd078/.tmp/A/d629d000efff44a6bcf81decd9e8717a 2023-04-15 02:56:18,257 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=20e761db5975fb9082f870bf056bd078, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:56:18,257 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34681] ipc.CallRunner(144): callId: 188 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:48664 deadline: 1681527438253, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=20e761db5975fb9082f870bf056bd078, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:18,257 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=20e761db5975fb9082f870bf056bd078, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:56:18,258 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34681] ipc.CallRunner(144): callId: 209 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:48586 deadline: 1681527438253, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=20e761db5975fb9082f870bf056bd078, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:18,382 INFO [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=364 (bloomFilter=true), to=hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/20e761db5975fb9082f870bf056bd078/.tmp/B/c692ce410fed4d8297e392cc7381eed0 2023-04-15 02:56:18,684 DEBUG [Thread-3440] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x08d3d8bc to 127.0.0.1:51762 2023-04-15 02:56:18,685 DEBUG [Thread-3440] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-15 02:56:18,685 DEBUG [Thread-3448] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x5467be78 to 127.0.0.1:51762 2023-04-15 02:56:18,685 DEBUG [Thread-3448] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-15 02:56:18,686 DEBUG [Thread-3446] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x49c34fbe to 127.0.0.1:51762 2023-04-15 02:56:18,686 DEBUG [Thread-3446] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-15 02:56:18,687 DEBUG [Thread-3442] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x7e6ad437 to 127.0.0.1:51762 2023-04-15 02:56:18,687 DEBUG [Thread-3442] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-15 02:56:18,689 DEBUG [Thread-3444] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x2f6f7187 to 127.0.0.1:51762 2023-04-15 02:56:18,689 DEBUG [Thread-3444] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-15 02:56:18,759 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=20e761db5975fb9082f870bf056bd078, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:56:18,759 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34681] ipc.CallRunner(144): callId: 190 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:48664 deadline: 1681527438759, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=20e761db5975fb9082f870bf056bd078, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:18,763 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34681] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=20e761db5975fb9082f870bf056bd078, server=jenkins-hbase20.apache.org,34681,1681527209133 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-15 02:56:18,764 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34681] ipc.CallRunner(144): callId: 211 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:48586 deadline: 1681527438763, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=20e761db5975fb9082f870bf056bd078, server=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:18,791 INFO [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=364 (bloomFilter=true), to=hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/20e761db5975fb9082f870bf056bd078/.tmp/C/d230775c80b84829871afc8a529d4cf9 2023-04-15 02:56:18,801 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/20e761db5975fb9082f870bf056bd078/.tmp/A/d629d000efff44a6bcf81decd9e8717a as hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/20e761db5975fb9082f870bf056bd078/A/d629d000efff44a6bcf81decd9e8717a 2023-04-15 02:56:18,805 INFO [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/20e761db5975fb9082f870bf056bd078/A/d629d000efff44a6bcf81decd9e8717a, entries=150, sequenceid=364, filesize=25.4 K 2023-04-15 02:56:18,806 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/20e761db5975fb9082f870bf056bd078/.tmp/B/c692ce410fed4d8297e392cc7381eed0 as hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/20e761db5975fb9082f870bf056bd078/B/c692ce410fed4d8297e392cc7381eed0 2023-04-15 02:56:18,811 INFO [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/20e761db5975fb9082f870bf056bd078/B/c692ce410fed4d8297e392cc7381eed0, entries=150, sequenceid=364, filesize=11.9 K 2023-04-15 02:56:18,811 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/20e761db5975fb9082f870bf056bd078/.tmp/C/d230775c80b84829871afc8a529d4cf9 as hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/20e761db5975fb9082f870bf056bd078/C/d230775c80b84829871afc8a529d4cf9 2023-04-15 02:56:18,815 INFO [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/20e761db5975fb9082f870bf056bd078/C/d230775c80b84829871afc8a529d4cf9, entries=150, sequenceid=364, filesize=11.9 K 2023-04-15 02:56:18,816 INFO [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=127.47 KB/130530 for 20e761db5975fb9082f870bf056bd078 in 1656ms, sequenceid=364, compaction requested=true 2023-04-15 02:56:18,816 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 20e761db5975fb9082f870bf056bd078: 2023-04-15 02:56:18,816 DEBUG [rs(jenkins-hbase20.apache.org,34681,1681527209133)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681527356526.20e761db5975fb9082f870bf056bd078. 2023-04-15 02:56:18,816 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-15 02:56:18,816 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-15 02:56:18,816 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:18,816 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-15 02:56:18,817 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,34681,1681527209133' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-15 02:56:18,818 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:18,818 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:56:18,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:18,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:56:18,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:56:18,818 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:56:18,818 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-15 02:56:18,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:56:18,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:56:18,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:56:18,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:18,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:56:18,820 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,34681,1681527209133' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-15 02:56:18,820 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1bff64fb[Count = 0] remaining members to acquire global barrier 2023-04-15 02:56:18,820 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-15 02:56:18,820 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:56:18,821 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:56:18,821 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:56:18,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:56:18,821 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-15 02:56:18,821 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-15 02:56:18,821 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,34681,1681527209133' in zk 2023-04-15 02:56:18,821 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:18,821 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-15 02:56:18,822 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-15 02:56:18,822 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-15 02:56:18,822 DEBUG [member: 'jenkins-hbase20.apache.org,34681,1681527209133' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-15 02:56:18,822 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:18,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:18,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:56:18,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:56:18,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:56:18,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:56:18,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:56:18,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:18,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:56:18,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:56:18,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:18,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,34681,1681527209133': 2023-04-15 02:56:18,825 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,34681,1681527209133' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-15 02:56:18,825 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-15 02:56:18,825 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-15 02:56:18,825 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-15 02:56:18,825 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-15 02:56:18,826 INFO [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-15 02:56:18,827 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:56:18,827 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:56:18,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:56:18,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-15 02:56:18,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-15 02:56:18,827 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:56:18,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:56:18,827 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-15 02:56:18,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-15 02:56:18,828 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-15 02:56:18,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-15 02:56:18,828 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:18,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:56:18,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:56:18,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-15 02:56:18,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:56:18,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:18,830 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:18,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-15 02:56:18,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-15 02:56:18,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:18,833 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-15 02:56:18,833 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:18,834 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-15 02:56:18,834 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-15 02:56:18,834 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-15 02:56:18,834 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-15 02:56:18,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-15 02:56:18,834 DEBUG [(jenkins-hbase20.apache.org,43619,1681527208212)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-15 02:56:18,834 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:56:18,834 DEBUG [Thread-3439] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-15 02:56:18,835 DEBUG [Thread-3439] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-15 02:56:18,834 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-15 02:56:18,834 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-15 02:56:18,835 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-15 02:56:18,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-15 02:56:18,835 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:18,835 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:56:18,836 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-15 02:56:18,836 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-15 02:56:18,935 DEBUG [Thread-3439] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-15 02:56:18,936 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-15 02:56:19,772 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34681] regionserver.HRegion(9158): Flush requested on 20e761db5975fb9082f870bf056bd078 2023-04-15 02:56:19,773 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 20e761db5975fb9082f870bf056bd078 3/3 column families, dataSize=134.18 KB heapSize=380.30 KB 2023-04-15 02:56:19,773 DEBUG [Thread-3431] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x4ae8f5b1 to 127.0.0.1:51762 2023-04-15 02:56:19,773 DEBUG [Thread-3431] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-15 02:56:19,774 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 20e761db5975fb9082f870bf056bd078, store=A 2023-04-15 02:56:19,774 DEBUG [Thread-3429] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x03b84625 to 127.0.0.1:51762 2023-04-15 02:56:19,774 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-15 02:56:19,774 DEBUG [Thread-3429] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-15 02:56:19,774 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 20e761db5975fb9082f870bf056bd078, store=B 2023-04-15 02:56:19,775 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-15 02:56:19,775 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 20e761db5975fb9082f870bf056bd078, store=C 2023-04-15 02:56:19,775 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-15 02:56:20,193 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230415be78ae3e463e48a2862ae704a1c157c3 to hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230415be78ae3e463e48a2862ae704a1c157c3 2023-04-15 02:56:20,603 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=401, memsize=47.0 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/20e761db5975fb9082f870bf056bd078/.tmp/A/051365f1a1d64f4293ad205aa06180ad 2023-04-15 02:56:21,022 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=401 (bloomFilter=true), to=hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/20e761db5975fb9082f870bf056bd078/.tmp/B/d6e5343bc5c241ae8406059a96e941b5 2023-04-15 02:56:21,437 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=401 (bloomFilter=true), to=hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/20e761db5975fb9082f870bf056bd078/.tmp/C/9f88097737a545c5bc685f2c98f3b22e 2023-04-15 02:56:21,442 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/20e761db5975fb9082f870bf056bd078/.tmp/A/051365f1a1d64f4293ad205aa06180ad as hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/20e761db5975fb9082f870bf056bd078/A/051365f1a1d64f4293ad205aa06180ad 2023-04-15 02:56:21,447 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/20e761db5975fb9082f870bf056bd078/A/051365f1a1d64f4293ad205aa06180ad, entries=150, sequenceid=401, filesize=25.4 K 2023-04-15 02:56:21,448 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/20e761db5975fb9082f870bf056bd078/.tmp/B/d6e5343bc5c241ae8406059a96e941b5 as hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/20e761db5975fb9082f870bf056bd078/B/d6e5343bc5c241ae8406059a96e941b5 2023-04-15 02:56:21,454 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/20e761db5975fb9082f870bf056bd078/B/d6e5343bc5c241ae8406059a96e941b5, entries=150, sequenceid=401, filesize=11.9 K 2023-04-15 02:56:21,455 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/20e761db5975fb9082f870bf056bd078/.tmp/C/9f88097737a545c5bc685f2c98f3b22e as hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/20e761db5975fb9082f870bf056bd078/C/9f88097737a545c5bc685f2c98f3b22e 2023-04-15 02:56:21,460 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/20e761db5975fb9082f870bf056bd078/C/9f88097737a545c5bc685f2c98f3b22e, entries=150, sequenceid=401, filesize=11.9 K 2023-04-15 02:56:21,461 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~140.89 KB/144270, heapSize ~400.17 KB/409776, currentSize=0 B/0 for 20e761db5975fb9082f870bf056bd078 in 1688ms, sequenceid=401, compaction requested=true 2023-04-15 02:56:21,461 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 20e761db5975fb9082f870bf056bd078: 2023-04-15 02:56:21,461 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-15 02:56:21,461 DEBUG [RS:0;jenkins-hbase20:34681-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 6 store files, 0 compacting, 6 eligible, 16 blocking 2023-04-15 02:56:21,461 DEBUG [RS:0;jenkins-hbase20:34681-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 6 store files, 0 compacting, 6 eligible, 16 blocking 2023-04-15 02:56:21,461 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-15 02:56:21,462 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-15 02:56:21,464 DEBUG [RS:0;jenkins-hbase20:34681-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 6 files of size 156822 starting at candidate #0 after considering 10 permutations with 10 in ratio 2023-04-15 02:56:21,464 DEBUG [RS:0;jenkins-hbase20:34681-shortCompactions-0] regionserver.HStore(1912): 20e761db5975fb9082f870bf056bd078/A is initiating minor compaction (all files) 2023-04-15 02:56:21,464 DEBUG [RS:0;jenkins-hbase20:34681-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 6 files of size 73662 starting at candidate #0 after considering 10 permutations with 10 in ratio 2023-04-15 02:56:21,464 INFO [RS:0;jenkins-hbase20:34681-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 20e761db5975fb9082f870bf056bd078/A in TestAcidGuarantees,,1681527356526.20e761db5975fb9082f870bf056bd078. 2023-04-15 02:56:21,464 DEBUG [RS:0;jenkins-hbase20:34681-longCompactions-0] regionserver.HStore(1912): 20e761db5975fb9082f870bf056bd078/B is initiating minor compaction (all files) 2023-04-15 02:56:21,465 INFO [RS:0;jenkins-hbase20:34681-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/20e761db5975fb9082f870bf056bd078/A/3825956b66294ff9b1219e17318fe437, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/20e761db5975fb9082f870bf056bd078/A/f7be98643a9b4774805a2ec65b40aa86, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/20e761db5975fb9082f870bf056bd078/A/1572e12eb1ac4642b76fca1232b4fd83, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/20e761db5975fb9082f870bf056bd078/A/17188db8659b466dbd46fce2f89d2b35, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/20e761db5975fb9082f870bf056bd078/A/d629d000efff44a6bcf81decd9e8717a, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/20e761db5975fb9082f870bf056bd078/A/051365f1a1d64f4293ad205aa06180ad] into tmpdir=hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/20e761db5975fb9082f870bf056bd078/.tmp, totalSize=153.1 K 2023-04-15 02:56:21,465 INFO [RS:0;jenkins-hbase20:34681-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 20e761db5975fb9082f870bf056bd078/B in TestAcidGuarantees,,1681527356526.20e761db5975fb9082f870bf056bd078. 2023-04-15 02:56:21,465 INFO [RS:0;jenkins-hbase20:34681-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/20e761db5975fb9082f870bf056bd078/B/770f8ea7fafd43e1b405486caae1c1bf, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/20e761db5975fb9082f870bf056bd078/B/706ef098332d4a7a9cafb947d28e285d, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/20e761db5975fb9082f870bf056bd078/B/6c7eee15ebd144d8b8b8d11efbf65068, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/20e761db5975fb9082f870bf056bd078/B/6abe126288384f4d92aac998836c31e0, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/20e761db5975fb9082f870bf056bd078/B/c692ce410fed4d8297e392cc7381eed0, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/20e761db5975fb9082f870bf056bd078/B/d6e5343bc5c241ae8406059a96e941b5] into tmpdir=hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/20e761db5975fb9082f870bf056bd078/.tmp, totalSize=71.9 K 2023-04-15 02:56:21,465 DEBUG [RS:0;jenkins-hbase20:34681-shortCompactions-0] compactions.Compactor(207): Compacting 3825956b66294ff9b1219e17318fe437, keycount=150, bloomtype=ROW, size=26.0 K, encoding=NONE, compression=NONE, seqNum=268, earliestPutTs=1681527372872 2023-04-15 02:56:21,465 DEBUG [RS:0;jenkins-hbase20:34681-longCompactions-0] compactions.Compactor(207): Compacting 770f8ea7fafd43e1b405486caae1c1bf, keycount=150, bloomtype=ROW, size=12.4 K, encoding=NONE, compression=NONE, seqNum=268, earliestPutTs=1681527372872 2023-04-15 02:56:21,466 DEBUG [RS:0;jenkins-hbase20:34681-shortCompactions-0] compactions.Compactor(207): Compacting f7be98643a9b4774805a2ec65b40aa86, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=305, earliestPutTs=1681527372957 2023-04-15 02:56:21,466 DEBUG [RS:0;jenkins-hbase20:34681-longCompactions-0] compactions.Compactor(207): Compacting 706ef098332d4a7a9cafb947d28e285d, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=305, earliestPutTs=1681527372957 2023-04-15 02:56:21,466 DEBUG [RS:0;jenkins-hbase20:34681-shortCompactions-0] compactions.Compactor(207): Compacting 1572e12eb1ac4642b76fca1232b4fd83, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=317, earliestPutTs=1681527373520 2023-04-15 02:56:21,466 DEBUG [RS:0;jenkins-hbase20:34681-longCompactions-0] compactions.Compactor(207): Compacting 6c7eee15ebd144d8b8b8d11efbf65068, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=317, earliestPutTs=1681527373520 2023-04-15 02:56:21,466 DEBUG [RS:0;jenkins-hbase20:34681-shortCompactions-0] compactions.Compactor(207): Compacting 17188db8659b466dbd46fce2f89d2b35, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=353, earliestPutTs=1681527375125 2023-04-15 02:56:21,467 DEBUG [RS:0;jenkins-hbase20:34681-longCompactions-0] compactions.Compactor(207): Compacting 6abe126288384f4d92aac998836c31e0, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=353, earliestPutTs=1681527375125 2023-04-15 02:56:21,467 DEBUG [RS:0;jenkins-hbase20:34681-shortCompactions-0] compactions.Compactor(207): Compacting d629d000efff44a6bcf81decd9e8717a, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=364, earliestPutTs=1681527375448 2023-04-15 02:56:21,467 DEBUG [RS:0;jenkins-hbase20:34681-longCompactions-0] compactions.Compactor(207): Compacting c692ce410fed4d8297e392cc7381eed0, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=364, earliestPutTs=1681527375448 2023-04-15 02:56:21,468 DEBUG [RS:0;jenkins-hbase20:34681-shortCompactions-0] compactions.Compactor(207): Compacting 051365f1a1d64f4293ad205aa06180ad, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=401, earliestPutTs=1681527377639 2023-04-15 02:56:21,468 DEBUG [RS:0;jenkins-hbase20:34681-longCompactions-0] compactions.Compactor(207): Compacting d6e5343bc5c241ae8406059a96e941b5, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=401, earliestPutTs=1681527377639 2023-04-15 02:56:21,483 INFO [RS:0;jenkins-hbase20:34681-longCompactions-0] throttle.PressureAwareThroughputController(145): 20e761db5975fb9082f870bf056bd078#B#compaction#506 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-15 02:56:21,488 INFO [RS:0;jenkins-hbase20:34681-shortCompactions-0] throttle.PressureAwareThroughputController(145): 20e761db5975fb9082f870bf056bd078#A#compaction#505 average throughput is 2.82 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-15 02:56:21,660 DEBUG [Thread-3437] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x5ec98d63 to 127.0.0.1:51762 2023-04-15 02:56:21,660 DEBUG [Thread-3437] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-15 02:56:21,690 DEBUG [Thread-3433] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x5ddb3e63 to 127.0.0.1:51762 2023-04-15 02:56:21,690 DEBUG [Thread-3433] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-15 02:56:21,695 DEBUG [Thread-3435] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x630ea023 to 127.0.0.1:51762 2023-04-15 02:56:21,695 DEBUG [Thread-3435] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-15 02:56:21,696 INFO [Listener at localhost.localdomain/33693] hbase.AcidGuaranteesTestTool(392): Finished test. Writers: 2023-04-15 02:56:21,696 INFO [Listener at localhost.localdomain/33693] hbase.AcidGuaranteesTestTool(394): wrote 108 2023-04-15 02:56:21,696 INFO [Listener at localhost.localdomain/33693] hbase.AcidGuaranteesTestTool(394): wrote 79 2023-04-15 02:56:21,696 INFO [Listener at localhost.localdomain/33693] hbase.AcidGuaranteesTestTool(394): wrote 44 2023-04-15 02:56:21,696 INFO [Listener at localhost.localdomain/33693] hbase.AcidGuaranteesTestTool(394): wrote 54 2023-04-15 02:56:21,696 INFO [Listener at localhost.localdomain/33693] hbase.AcidGuaranteesTestTool(394): wrote 45 2023-04-15 02:56:21,696 INFO [Listener at localhost.localdomain/33693] hbase.AcidGuaranteesTestTool(396): Readers: 2023-04-15 02:56:21,696 INFO [Listener at localhost.localdomain/33693] hbase.AcidGuaranteesTestTool(398): read 5489 2023-04-15 02:56:21,697 INFO [Listener at localhost.localdomain/33693] hbase.AcidGuaranteesTestTool(398): read 5407 2023-04-15 02:56:21,697 INFO [Listener at localhost.localdomain/33693] hbase.AcidGuaranteesTestTool(398): read 5435 2023-04-15 02:56:21,697 INFO [Listener at localhost.localdomain/33693] hbase.AcidGuaranteesTestTool(398): read 5485 2023-04-15 02:56:21,697 INFO [Listener at localhost.localdomain/33693] hbase.AcidGuaranteesTestTool(398): read 5405 2023-04-15 02:56:21,697 INFO [Listener at localhost.localdomain/33693] hbase.AcidGuaranteesTestTool(400): Scanners: 2023-04-15 02:56:21,697 INFO [Listener at localhost.localdomain/33693] client.ConnectionImplementation(1974): Closing master protocol: MasterService 2023-04-15 02:56:21,697 DEBUG [Listener at localhost.localdomain/33693] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x65a9ccaf to 127.0.0.1:51762 2023-04-15 02:56:21,697 DEBUG [Listener at localhost.localdomain/33693] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-15 02:56:21,698 INFO [Listener at localhost.localdomain/33693] client.HBaseAdmin$15(890): Started disable of TestAcidGuarantees 2023-04-15 02:56:21,698 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.HMaster$11(2418): Client=jenkins//148.251.75.209 disable TestAcidGuarantees 2023-04-15 02:56:21,699 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] procedure2.ProcedureExecutor(1029): Stored pid=62, state=RUNNABLE:DISABLE_TABLE_PREPARE; DisableTableProcedure table=TestAcidGuarantees 2023-04-15 02:56:21,702 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(1227): Checking to see if procedure is done pid=62 2023-04-15 02:56:21,703 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1681527381703"}]},"ts":"1681527381703"} 2023-04-15 02:56:21,704 INFO [PEWorker-3] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=DISABLING in hbase:meta 2023-04-15 02:56:21,712 INFO [PEWorker-3] procedure.DisableTableProcedure(293): Set TestAcidGuarantees to state=DISABLING 2023-04-15 02:56:21,713 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=63, ppid=62, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=20e761db5975fb9082f870bf056bd078, UNASSIGN}] 2023-04-15 02:56:21,714 INFO [PEWorker-3] procedure.MasterProcedureScheduler(727): Took xlock for pid=63, ppid=62, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=20e761db5975fb9082f870bf056bd078, UNASSIGN 2023-04-15 02:56:21,715 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=63 updating hbase:meta row=20e761db5975fb9082f870bf056bd078, regionState=CLOSING, regionLocation=jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:21,715 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1681527356526.20e761db5975fb9082f870bf056bd078.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1681527381715"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1681527381715"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1681527381715"}]},"ts":"1681527381715"} 2023-04-15 02:56:21,717 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=64, ppid=63, state=RUNNABLE; CloseRegionProcedure 20e761db5975fb9082f870bf056bd078, server=jenkins-hbase20.apache.org,34681,1681527209133}] 2023-04-15 02:56:21,803 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(1227): Checking to see if procedure is done pid=62 2023-04-15 02:56:21,869 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] handler.UnassignRegionHandler(109): Close 20e761db5975fb9082f870bf056bd078 2023-04-15 02:56:21,869 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1604): Closing 20e761db5975fb9082f870bf056bd078, disabling compactions & flushes 2023-04-15 02:56:21,869 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1858): waiting for 2 compactions to complete for region TestAcidGuarantees,,1681527356526.20e761db5975fb9082f870bf056bd078. 2023-04-15 02:56:21,893 DEBUG [RS:0;jenkins-hbase20:34681-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/20e761db5975fb9082f870bf056bd078/.tmp/B/d6057730ef64497da7dda5ab90dc2c30 as hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/20e761db5975fb9082f870bf056bd078/B/d6057730ef64497da7dda5ab90dc2c30 2023-04-15 02:56:21,897 INFO [RS:0;jenkins-hbase20:34681-longCompactions-0] regionserver.HStore(1652): Completed compaction of 6 (all) file(s) in 20e761db5975fb9082f870bf056bd078/B of 20e761db5975fb9082f870bf056bd078 into d6057730ef64497da7dda5ab90dc2c30(size=12.6 K), total size for store is 12.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-15 02:56:21,897 DEBUG [RS:0;jenkins-hbase20:34681-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 20e761db5975fb9082f870bf056bd078: 2023-04-15 02:56:21,897 INFO [RS:0;jenkins-hbase20:34681-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681527356526.20e761db5975fb9082f870bf056bd078., storeName=20e761db5975fb9082f870bf056bd078/B, priority=10, startTime=1681527381461; duration=0sec 2023-04-15 02:56:21,897 DEBUG [RS:0;jenkins-hbase20:34681-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-15 02:56:21,897 DEBUG [RS:0;jenkins-hbase20:34681-longCompactions-0] regionserver.CompactSplit(415): Not compacting TestAcidGuarantees,,1681527356526.20e761db5975fb9082f870bf056bd078. because compaction request was cancelled 2023-04-15 02:56:22,005 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(1227): Checking to see if procedure is done pid=62 2023-04-15 02:56:22,307 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(1227): Checking to see if procedure is done pid=62 2023-04-15 02:56:22,713 DEBUG [RS:0;jenkins-hbase20:34681-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/20e761db5975fb9082f870bf056bd078/.tmp/A/44475d30540b4839a6d78e65f3b7ce28 as hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/20e761db5975fb9082f870bf056bd078/A/44475d30540b4839a6d78e65f3b7ce28 2023-04-15 02:56:22,720 INFO [RS:0;jenkins-hbase20:34681-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 6 (all) file(s) in 20e761db5975fb9082f870bf056bd078/A of 20e761db5975fb9082f870bf056bd078 into 44475d30540b4839a6d78e65f3b7ce28(size=26.2 K), total size for store is 26.2 K. This selection was in queue for 0sec, and took 1sec to execute. 2023-04-15 02:56:22,720 DEBUG [RS:0;jenkins-hbase20:34681-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 20e761db5975fb9082f870bf056bd078: 2023-04-15 02:56:22,720 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1681527356526.20e761db5975fb9082f870bf056bd078. 2023-04-15 02:56:22,720 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1681527356526.20e761db5975fb9082f870bf056bd078. 2023-04-15 02:56:22,720 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1681527356526.20e761db5975fb9082f870bf056bd078. after waiting 0 ms 2023-04-15 02:56:22,720 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1681527356526.20e761db5975fb9082f870bf056bd078. 2023-04-15 02:56:22,720 INFO [RS:0;jenkins-hbase20:34681-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681527356526.20e761db5975fb9082f870bf056bd078., storeName=20e761db5975fb9082f870bf056bd078/A, priority=10, startTime=1681527381461; duration=1sec 2023-04-15 02:56:22,721 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(2745): Flushing 20e761db5975fb9082f870bf056bd078 3/3 column families, dataSize=20.13 KB heapSize=60.52 KB 2023-04-15 02:56:22,721 DEBUG [RS:0;jenkins-hbase20:34681-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-15 02:56:22,721 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 20e761db5975fb9082f870bf056bd078, store=A 2023-04-15 02:56:22,721 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-15 02:56:22,721 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 20e761db5975fb9082f870bf056bd078, store=B 2023-04-15 02:56:22,721 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-15 02:56:22,721 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 20e761db5975fb9082f870bf056bd078, store=C 2023-04-15 02:56:22,721 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-15 02:56:22,809 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(1227): Checking to see if procedure is done pid=62 2023-04-15 02:56:23,143 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023041507c813b405274e1e90689e5288332391 to hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023041507c813b405274e1e90689e5288332391 2023-04-15 02:56:23,550 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=409, memsize=6.7 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/20e761db5975fb9082f870bf056bd078/.tmp/A/f8cd1ab909254accb88a7e0be3ab020b 2023-04-15 02:56:23,810 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(1227): Checking to see if procedure is done pid=62 2023-04-15 02:56:23,965 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=6.71 KB at sequenceid=409 (bloomFilter=true), to=hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/20e761db5975fb9082f870bf056bd078/.tmp/B/bad5e513a0fe40ae84e0f11a25ead334 2023-04-15 02:56:24,382 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=6.71 KB at sequenceid=409 (bloomFilter=true), to=hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/20e761db5975fb9082f870bf056bd078/.tmp/C/2af9856544734ee3a41835a2bcc002e1 2023-04-15 02:56:24,385 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/20e761db5975fb9082f870bf056bd078/.tmp/A/f8cd1ab909254accb88a7e0be3ab020b as hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/20e761db5975fb9082f870bf056bd078/A/f8cd1ab909254accb88a7e0be3ab020b 2023-04-15 02:56:24,388 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/20e761db5975fb9082f870bf056bd078/A/f8cd1ab909254accb88a7e0be3ab020b, entries=100, sequenceid=409, filesize=18.6 K 2023-04-15 02:56:24,389 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/20e761db5975fb9082f870bf056bd078/.tmp/B/bad5e513a0fe40ae84e0f11a25ead334 as hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/20e761db5975fb9082f870bf056bd078/B/bad5e513a0fe40ae84e0f11a25ead334 2023-04-15 02:56:24,391 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/20e761db5975fb9082f870bf056bd078/B/bad5e513a0fe40ae84e0f11a25ead334, entries=100, sequenceid=409, filesize=9.5 K 2023-04-15 02:56:24,392 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/20e761db5975fb9082f870bf056bd078/.tmp/C/2af9856544734ee3a41835a2bcc002e1 as hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/20e761db5975fb9082f870bf056bd078/C/2af9856544734ee3a41835a2bcc002e1 2023-04-15 02:56:24,394 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/20e761db5975fb9082f870bf056bd078/C/2af9856544734ee3a41835a2bcc002e1, entries=100, sequenceid=409, filesize=9.5 K 2023-04-15 02:56:24,395 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(2948): Finished flush of dataSize ~20.13 KB/20610, heapSize ~60.47 KB/61920, currentSize=0 B/0 for 20e761db5975fb9082f870bf056bd078 in 1675ms, sequenceid=409, compaction requested=true 2023-04-15 02:56:24,412 DEBUG [StoreCloser-TestAcidGuarantees,,1681527356526.20e761db5975fb9082f870bf056bd078.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/20e761db5975fb9082f870bf056bd078/A/a714ed51fc7c46378b9e8630cea6f95f, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/20e761db5975fb9082f870bf056bd078/A/78ddfefdb20c4498b0be0f13b8b9bf97, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/20e761db5975fb9082f870bf056bd078/A/7f2af62291f44d66a5b3f7a3ff162745, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/20e761db5975fb9082f870bf056bd078/A/3e28da5a132b4b5e9abf7f7e92c7bde4, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/20e761db5975fb9082f870bf056bd078/A/682a33ab364b4412a3f05a564e0c8208, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/20e761db5975fb9082f870bf056bd078/A/61964aa8922847d490712f3d4966e25c, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/20e761db5975fb9082f870bf056bd078/A/1839c025181b488a9f7e760e76b34ff6, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/20e761db5975fb9082f870bf056bd078/A/0312c634961f48589caed89d0a1783c0, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/20e761db5975fb9082f870bf056bd078/A/835c5281097746cb9ba12cc89ec8adbe, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/20e761db5975fb9082f870bf056bd078/A/168e7843b8c9470f857fedbb03a0eb8c, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/20e761db5975fb9082f870bf056bd078/A/3536d0084b564ccca286193f0c26d057, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/20e761db5975fb9082f870bf056bd078/A/88fc7b8e881e4037b4ce9aadb3c830c6, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/20e761db5975fb9082f870bf056bd078/A/ad3786b6e53e4004823e008fe6273198, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/20e761db5975fb9082f870bf056bd078/A/a59ed8a7c4bd40a6879c548ffb1870d2, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/20e761db5975fb9082f870bf056bd078/A/8c4dc0ae326943b4ae95dc85a0dbc4e5, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/20e761db5975fb9082f870bf056bd078/A/7ff283029d7748108529cd72184d2d13, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/20e761db5975fb9082f870bf056bd078/A/3825956b66294ff9b1219e17318fe437, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/20e761db5975fb9082f870bf056bd078/A/f7be98643a9b4774805a2ec65b40aa86, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/20e761db5975fb9082f870bf056bd078/A/1572e12eb1ac4642b76fca1232b4fd83, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/20e761db5975fb9082f870bf056bd078/A/17188db8659b466dbd46fce2f89d2b35, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/20e761db5975fb9082f870bf056bd078/A/d629d000efff44a6bcf81decd9e8717a, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/20e761db5975fb9082f870bf056bd078/A/051365f1a1d64f4293ad205aa06180ad] to archive 2023-04-15 02:56:24,413 DEBUG [StoreCloser-TestAcidGuarantees,,1681527356526.20e761db5975fb9082f870bf056bd078.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-15 02:56:24,414 DEBUG [StoreCloser-TestAcidGuarantees,,1681527356526.20e761db5975fb9082f870bf056bd078.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/20e761db5975fb9082f870bf056bd078/A/a714ed51fc7c46378b9e8630cea6f95f to hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/archive/data/default/TestAcidGuarantees/20e761db5975fb9082f870bf056bd078/A/a714ed51fc7c46378b9e8630cea6f95f 2023-04-15 02:56:24,415 DEBUG [StoreCloser-TestAcidGuarantees,,1681527356526.20e761db5975fb9082f870bf056bd078.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/20e761db5975fb9082f870bf056bd078/A/78ddfefdb20c4498b0be0f13b8b9bf97 to hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/archive/data/default/TestAcidGuarantees/20e761db5975fb9082f870bf056bd078/A/78ddfefdb20c4498b0be0f13b8b9bf97 2023-04-15 02:56:24,416 DEBUG [StoreCloser-TestAcidGuarantees,,1681527356526.20e761db5975fb9082f870bf056bd078.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/20e761db5975fb9082f870bf056bd078/A/7f2af62291f44d66a5b3f7a3ff162745 to hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/archive/data/default/TestAcidGuarantees/20e761db5975fb9082f870bf056bd078/A/7f2af62291f44d66a5b3f7a3ff162745 2023-04-15 02:56:24,416 DEBUG [StoreCloser-TestAcidGuarantees,,1681527356526.20e761db5975fb9082f870bf056bd078.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/20e761db5975fb9082f870bf056bd078/A/3e28da5a132b4b5e9abf7f7e92c7bde4 to hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/archive/data/default/TestAcidGuarantees/20e761db5975fb9082f870bf056bd078/A/3e28da5a132b4b5e9abf7f7e92c7bde4 2023-04-15 02:56:24,417 DEBUG [StoreCloser-TestAcidGuarantees,,1681527356526.20e761db5975fb9082f870bf056bd078.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/20e761db5975fb9082f870bf056bd078/A/682a33ab364b4412a3f05a564e0c8208 to hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/archive/data/default/TestAcidGuarantees/20e761db5975fb9082f870bf056bd078/A/682a33ab364b4412a3f05a564e0c8208 2023-04-15 02:56:24,418 DEBUG [StoreCloser-TestAcidGuarantees,,1681527356526.20e761db5975fb9082f870bf056bd078.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/20e761db5975fb9082f870bf056bd078/A/61964aa8922847d490712f3d4966e25c to hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/archive/data/default/TestAcidGuarantees/20e761db5975fb9082f870bf056bd078/A/61964aa8922847d490712f3d4966e25c 2023-04-15 02:56:24,419 DEBUG [StoreCloser-TestAcidGuarantees,,1681527356526.20e761db5975fb9082f870bf056bd078.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/20e761db5975fb9082f870bf056bd078/A/1839c025181b488a9f7e760e76b34ff6 to hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/archive/data/default/TestAcidGuarantees/20e761db5975fb9082f870bf056bd078/A/1839c025181b488a9f7e760e76b34ff6 2023-04-15 02:56:24,420 DEBUG [StoreCloser-TestAcidGuarantees,,1681527356526.20e761db5975fb9082f870bf056bd078.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/20e761db5975fb9082f870bf056bd078/A/0312c634961f48589caed89d0a1783c0 to hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/archive/data/default/TestAcidGuarantees/20e761db5975fb9082f870bf056bd078/A/0312c634961f48589caed89d0a1783c0 2023-04-15 02:56:24,420 DEBUG [StoreCloser-TestAcidGuarantees,,1681527356526.20e761db5975fb9082f870bf056bd078.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/20e761db5975fb9082f870bf056bd078/A/835c5281097746cb9ba12cc89ec8adbe to hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/archive/data/default/TestAcidGuarantees/20e761db5975fb9082f870bf056bd078/A/835c5281097746cb9ba12cc89ec8adbe 2023-04-15 02:56:24,421 DEBUG [StoreCloser-TestAcidGuarantees,,1681527356526.20e761db5975fb9082f870bf056bd078.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/20e761db5975fb9082f870bf056bd078/A/168e7843b8c9470f857fedbb03a0eb8c to hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/archive/data/default/TestAcidGuarantees/20e761db5975fb9082f870bf056bd078/A/168e7843b8c9470f857fedbb03a0eb8c 2023-04-15 02:56:24,422 DEBUG [StoreCloser-TestAcidGuarantees,,1681527356526.20e761db5975fb9082f870bf056bd078.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/20e761db5975fb9082f870bf056bd078/A/3536d0084b564ccca286193f0c26d057 to hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/archive/data/default/TestAcidGuarantees/20e761db5975fb9082f870bf056bd078/A/3536d0084b564ccca286193f0c26d057 2023-04-15 02:56:24,423 DEBUG [StoreCloser-TestAcidGuarantees,,1681527356526.20e761db5975fb9082f870bf056bd078.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/20e761db5975fb9082f870bf056bd078/A/88fc7b8e881e4037b4ce9aadb3c830c6 to hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/archive/data/default/TestAcidGuarantees/20e761db5975fb9082f870bf056bd078/A/88fc7b8e881e4037b4ce9aadb3c830c6 2023-04-15 02:56:24,424 DEBUG [StoreCloser-TestAcidGuarantees,,1681527356526.20e761db5975fb9082f870bf056bd078.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/20e761db5975fb9082f870bf056bd078/A/ad3786b6e53e4004823e008fe6273198 to hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/archive/data/default/TestAcidGuarantees/20e761db5975fb9082f870bf056bd078/A/ad3786b6e53e4004823e008fe6273198 2023-04-15 02:56:24,424 DEBUG [StoreCloser-TestAcidGuarantees,,1681527356526.20e761db5975fb9082f870bf056bd078.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/20e761db5975fb9082f870bf056bd078/A/a59ed8a7c4bd40a6879c548ffb1870d2 to hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/archive/data/default/TestAcidGuarantees/20e761db5975fb9082f870bf056bd078/A/a59ed8a7c4bd40a6879c548ffb1870d2 2023-04-15 02:56:24,425 DEBUG [StoreCloser-TestAcidGuarantees,,1681527356526.20e761db5975fb9082f870bf056bd078.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/20e761db5975fb9082f870bf056bd078/A/8c4dc0ae326943b4ae95dc85a0dbc4e5 to hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/archive/data/default/TestAcidGuarantees/20e761db5975fb9082f870bf056bd078/A/8c4dc0ae326943b4ae95dc85a0dbc4e5 2023-04-15 02:56:24,426 DEBUG [StoreCloser-TestAcidGuarantees,,1681527356526.20e761db5975fb9082f870bf056bd078.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/20e761db5975fb9082f870bf056bd078/A/7ff283029d7748108529cd72184d2d13 to hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/archive/data/default/TestAcidGuarantees/20e761db5975fb9082f870bf056bd078/A/7ff283029d7748108529cd72184d2d13 2023-04-15 02:56:24,427 DEBUG [StoreCloser-TestAcidGuarantees,,1681527356526.20e761db5975fb9082f870bf056bd078.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/20e761db5975fb9082f870bf056bd078/A/3825956b66294ff9b1219e17318fe437 to hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/archive/data/default/TestAcidGuarantees/20e761db5975fb9082f870bf056bd078/A/3825956b66294ff9b1219e17318fe437 2023-04-15 02:56:24,428 DEBUG [StoreCloser-TestAcidGuarantees,,1681527356526.20e761db5975fb9082f870bf056bd078.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/20e761db5975fb9082f870bf056bd078/A/f7be98643a9b4774805a2ec65b40aa86 to hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/archive/data/default/TestAcidGuarantees/20e761db5975fb9082f870bf056bd078/A/f7be98643a9b4774805a2ec65b40aa86 2023-04-15 02:56:24,428 DEBUG [StoreCloser-TestAcidGuarantees,,1681527356526.20e761db5975fb9082f870bf056bd078.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/20e761db5975fb9082f870bf056bd078/A/1572e12eb1ac4642b76fca1232b4fd83 to hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/archive/data/default/TestAcidGuarantees/20e761db5975fb9082f870bf056bd078/A/1572e12eb1ac4642b76fca1232b4fd83 2023-04-15 02:56:24,429 DEBUG [StoreCloser-TestAcidGuarantees,,1681527356526.20e761db5975fb9082f870bf056bd078.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/20e761db5975fb9082f870bf056bd078/A/17188db8659b466dbd46fce2f89d2b35 to hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/archive/data/default/TestAcidGuarantees/20e761db5975fb9082f870bf056bd078/A/17188db8659b466dbd46fce2f89d2b35 2023-04-15 02:56:24,430 DEBUG [StoreCloser-TestAcidGuarantees,,1681527356526.20e761db5975fb9082f870bf056bd078.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/20e761db5975fb9082f870bf056bd078/A/d629d000efff44a6bcf81decd9e8717a to hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/archive/data/default/TestAcidGuarantees/20e761db5975fb9082f870bf056bd078/A/d629d000efff44a6bcf81decd9e8717a 2023-04-15 02:56:24,431 DEBUG [StoreCloser-TestAcidGuarantees,,1681527356526.20e761db5975fb9082f870bf056bd078.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/20e761db5975fb9082f870bf056bd078/A/051365f1a1d64f4293ad205aa06180ad to hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/archive/data/default/TestAcidGuarantees/20e761db5975fb9082f870bf056bd078/A/051365f1a1d64f4293ad205aa06180ad 2023-04-15 02:56:24,434 DEBUG [StoreCloser-TestAcidGuarantees,,1681527356526.20e761db5975fb9082f870bf056bd078.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/20e761db5975fb9082f870bf056bd078/B/ce7e96538ce54fc2bed1e90c526b8f0e, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/20e761db5975fb9082f870bf056bd078/B/a78ad28d40df4cb39afa5886b9b0cf03, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/20e761db5975fb9082f870bf056bd078/B/0969d679f2bb4597bac256d78f824067, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/20e761db5975fb9082f870bf056bd078/B/cc033b3c9f6e443b9aa13784606e91be, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/20e761db5975fb9082f870bf056bd078/B/0c295b6fa6c34914ba5164667243e923, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/20e761db5975fb9082f870bf056bd078/B/c030a2631f8f4bc0b69a4c216dc58a93, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/20e761db5975fb9082f870bf056bd078/B/f092b0e0f6b44cb280f40a09566e7c43, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/20e761db5975fb9082f870bf056bd078/B/d3cc26ffda714b4786f1a1389190f5d5, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/20e761db5975fb9082f870bf056bd078/B/13265909e92d49d793b4cf103df530a3, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/20e761db5975fb9082f870bf056bd078/B/06e500fe23174a8896b3d0f9dd3a4863, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/20e761db5975fb9082f870bf056bd078/B/d4b604a3725b405d939d12f478ff880c, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/20e761db5975fb9082f870bf056bd078/B/d82948cce74045688628e15cec64e1fd, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/20e761db5975fb9082f870bf056bd078/B/a7e7f41bd36a4aefbcf65317d8aa8ab8, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/20e761db5975fb9082f870bf056bd078/B/579e24eeec284cf2ae86663ebdf3335f, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/20e761db5975fb9082f870bf056bd078/B/a48b06a3343d435b86de97231507762d, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/20e761db5975fb9082f870bf056bd078/B/770f8ea7fafd43e1b405486caae1c1bf, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/20e761db5975fb9082f870bf056bd078/B/82721bd5833e4862be0e0abd0b57cde5, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/20e761db5975fb9082f870bf056bd078/B/706ef098332d4a7a9cafb947d28e285d, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/20e761db5975fb9082f870bf056bd078/B/6c7eee15ebd144d8b8b8d11efbf65068, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/20e761db5975fb9082f870bf056bd078/B/6abe126288384f4d92aac998836c31e0, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/20e761db5975fb9082f870bf056bd078/B/c692ce410fed4d8297e392cc7381eed0, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/20e761db5975fb9082f870bf056bd078/B/d6e5343bc5c241ae8406059a96e941b5] to archive 2023-04-15 02:56:24,434 DEBUG [StoreCloser-TestAcidGuarantees,,1681527356526.20e761db5975fb9082f870bf056bd078.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-15 02:56:24,435 DEBUG [StoreCloser-TestAcidGuarantees,,1681527356526.20e761db5975fb9082f870bf056bd078.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/20e761db5975fb9082f870bf056bd078/B/ce7e96538ce54fc2bed1e90c526b8f0e to hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/archive/data/default/TestAcidGuarantees/20e761db5975fb9082f870bf056bd078/B/ce7e96538ce54fc2bed1e90c526b8f0e 2023-04-15 02:56:24,436 DEBUG [StoreCloser-TestAcidGuarantees,,1681527356526.20e761db5975fb9082f870bf056bd078.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/20e761db5975fb9082f870bf056bd078/B/a78ad28d40df4cb39afa5886b9b0cf03 to hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/archive/data/default/TestAcidGuarantees/20e761db5975fb9082f870bf056bd078/B/a78ad28d40df4cb39afa5886b9b0cf03 2023-04-15 02:56:24,437 DEBUG [StoreCloser-TestAcidGuarantees,,1681527356526.20e761db5975fb9082f870bf056bd078.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/20e761db5975fb9082f870bf056bd078/B/0969d679f2bb4597bac256d78f824067 to hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/archive/data/default/TestAcidGuarantees/20e761db5975fb9082f870bf056bd078/B/0969d679f2bb4597bac256d78f824067 2023-04-15 02:56:24,438 DEBUG [StoreCloser-TestAcidGuarantees,,1681527356526.20e761db5975fb9082f870bf056bd078.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/20e761db5975fb9082f870bf056bd078/B/cc033b3c9f6e443b9aa13784606e91be to hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/archive/data/default/TestAcidGuarantees/20e761db5975fb9082f870bf056bd078/B/cc033b3c9f6e443b9aa13784606e91be 2023-04-15 02:56:24,439 DEBUG [StoreCloser-TestAcidGuarantees,,1681527356526.20e761db5975fb9082f870bf056bd078.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/20e761db5975fb9082f870bf056bd078/B/0c295b6fa6c34914ba5164667243e923 to hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/archive/data/default/TestAcidGuarantees/20e761db5975fb9082f870bf056bd078/B/0c295b6fa6c34914ba5164667243e923 2023-04-15 02:56:24,440 DEBUG [StoreCloser-TestAcidGuarantees,,1681527356526.20e761db5975fb9082f870bf056bd078.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/20e761db5975fb9082f870bf056bd078/B/c030a2631f8f4bc0b69a4c216dc58a93 to hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/archive/data/default/TestAcidGuarantees/20e761db5975fb9082f870bf056bd078/B/c030a2631f8f4bc0b69a4c216dc58a93 2023-04-15 02:56:24,441 DEBUG [StoreCloser-TestAcidGuarantees,,1681527356526.20e761db5975fb9082f870bf056bd078.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/20e761db5975fb9082f870bf056bd078/B/f092b0e0f6b44cb280f40a09566e7c43 to hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/archive/data/default/TestAcidGuarantees/20e761db5975fb9082f870bf056bd078/B/f092b0e0f6b44cb280f40a09566e7c43 2023-04-15 02:56:24,442 DEBUG [StoreCloser-TestAcidGuarantees,,1681527356526.20e761db5975fb9082f870bf056bd078.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/20e761db5975fb9082f870bf056bd078/B/d3cc26ffda714b4786f1a1389190f5d5 to hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/archive/data/default/TestAcidGuarantees/20e761db5975fb9082f870bf056bd078/B/d3cc26ffda714b4786f1a1389190f5d5 2023-04-15 02:56:24,443 DEBUG [StoreCloser-TestAcidGuarantees,,1681527356526.20e761db5975fb9082f870bf056bd078.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/20e761db5975fb9082f870bf056bd078/B/13265909e92d49d793b4cf103df530a3 to hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/archive/data/default/TestAcidGuarantees/20e761db5975fb9082f870bf056bd078/B/13265909e92d49d793b4cf103df530a3 2023-04-15 02:56:24,444 DEBUG [StoreCloser-TestAcidGuarantees,,1681527356526.20e761db5975fb9082f870bf056bd078.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/20e761db5975fb9082f870bf056bd078/B/06e500fe23174a8896b3d0f9dd3a4863 to hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/archive/data/default/TestAcidGuarantees/20e761db5975fb9082f870bf056bd078/B/06e500fe23174a8896b3d0f9dd3a4863 2023-04-15 02:56:24,445 DEBUG [StoreCloser-TestAcidGuarantees,,1681527356526.20e761db5975fb9082f870bf056bd078.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/20e761db5975fb9082f870bf056bd078/B/d4b604a3725b405d939d12f478ff880c to hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/archive/data/default/TestAcidGuarantees/20e761db5975fb9082f870bf056bd078/B/d4b604a3725b405d939d12f478ff880c 2023-04-15 02:56:24,447 DEBUG [StoreCloser-TestAcidGuarantees,,1681527356526.20e761db5975fb9082f870bf056bd078.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/20e761db5975fb9082f870bf056bd078/B/d82948cce74045688628e15cec64e1fd to hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/archive/data/default/TestAcidGuarantees/20e761db5975fb9082f870bf056bd078/B/d82948cce74045688628e15cec64e1fd 2023-04-15 02:56:24,448 DEBUG [StoreCloser-TestAcidGuarantees,,1681527356526.20e761db5975fb9082f870bf056bd078.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/20e761db5975fb9082f870bf056bd078/B/a7e7f41bd36a4aefbcf65317d8aa8ab8 to hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/archive/data/default/TestAcidGuarantees/20e761db5975fb9082f870bf056bd078/B/a7e7f41bd36a4aefbcf65317d8aa8ab8 2023-04-15 02:56:24,449 DEBUG [StoreCloser-TestAcidGuarantees,,1681527356526.20e761db5975fb9082f870bf056bd078.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/20e761db5975fb9082f870bf056bd078/B/579e24eeec284cf2ae86663ebdf3335f to hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/archive/data/default/TestAcidGuarantees/20e761db5975fb9082f870bf056bd078/B/579e24eeec284cf2ae86663ebdf3335f 2023-04-15 02:56:24,450 DEBUG [StoreCloser-TestAcidGuarantees,,1681527356526.20e761db5975fb9082f870bf056bd078.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/20e761db5975fb9082f870bf056bd078/B/a48b06a3343d435b86de97231507762d to hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/archive/data/default/TestAcidGuarantees/20e761db5975fb9082f870bf056bd078/B/a48b06a3343d435b86de97231507762d 2023-04-15 02:56:24,451 DEBUG [StoreCloser-TestAcidGuarantees,,1681527356526.20e761db5975fb9082f870bf056bd078.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/20e761db5975fb9082f870bf056bd078/B/770f8ea7fafd43e1b405486caae1c1bf to hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/archive/data/default/TestAcidGuarantees/20e761db5975fb9082f870bf056bd078/B/770f8ea7fafd43e1b405486caae1c1bf 2023-04-15 02:56:24,452 DEBUG [StoreCloser-TestAcidGuarantees,,1681527356526.20e761db5975fb9082f870bf056bd078.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/20e761db5975fb9082f870bf056bd078/B/82721bd5833e4862be0e0abd0b57cde5 to hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/archive/data/default/TestAcidGuarantees/20e761db5975fb9082f870bf056bd078/B/82721bd5833e4862be0e0abd0b57cde5 2023-04-15 02:56:24,453 DEBUG [StoreCloser-TestAcidGuarantees,,1681527356526.20e761db5975fb9082f870bf056bd078.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/20e761db5975fb9082f870bf056bd078/B/706ef098332d4a7a9cafb947d28e285d to hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/archive/data/default/TestAcidGuarantees/20e761db5975fb9082f870bf056bd078/B/706ef098332d4a7a9cafb947d28e285d 2023-04-15 02:56:24,453 DEBUG [StoreCloser-TestAcidGuarantees,,1681527356526.20e761db5975fb9082f870bf056bd078.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/20e761db5975fb9082f870bf056bd078/B/6c7eee15ebd144d8b8b8d11efbf65068 to hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/archive/data/default/TestAcidGuarantees/20e761db5975fb9082f870bf056bd078/B/6c7eee15ebd144d8b8b8d11efbf65068 2023-04-15 02:56:24,454 DEBUG [StoreCloser-TestAcidGuarantees,,1681527356526.20e761db5975fb9082f870bf056bd078.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/20e761db5975fb9082f870bf056bd078/B/6abe126288384f4d92aac998836c31e0 to hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/archive/data/default/TestAcidGuarantees/20e761db5975fb9082f870bf056bd078/B/6abe126288384f4d92aac998836c31e0 2023-04-15 02:56:24,455 DEBUG [StoreCloser-TestAcidGuarantees,,1681527356526.20e761db5975fb9082f870bf056bd078.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/20e761db5975fb9082f870bf056bd078/B/c692ce410fed4d8297e392cc7381eed0 to hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/archive/data/default/TestAcidGuarantees/20e761db5975fb9082f870bf056bd078/B/c692ce410fed4d8297e392cc7381eed0 2023-04-15 02:56:24,456 DEBUG [StoreCloser-TestAcidGuarantees,,1681527356526.20e761db5975fb9082f870bf056bd078.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/20e761db5975fb9082f870bf056bd078/B/d6e5343bc5c241ae8406059a96e941b5 to hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/archive/data/default/TestAcidGuarantees/20e761db5975fb9082f870bf056bd078/B/d6e5343bc5c241ae8406059a96e941b5 2023-04-15 02:56:24,463 DEBUG [StoreCloser-TestAcidGuarantees,,1681527356526.20e761db5975fb9082f870bf056bd078.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/20e761db5975fb9082f870bf056bd078/C/bc0770e966664656b1a39dc2b7bc02ad, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/20e761db5975fb9082f870bf056bd078/C/3d55dce025b5415982c8518bb01bd7d4, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/20e761db5975fb9082f870bf056bd078/C/6dd346ee5c4e4a1f8bf4d4dff5b0287c, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/20e761db5975fb9082f870bf056bd078/C/43db6e1add50473786a2b33dee3985cd, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/20e761db5975fb9082f870bf056bd078/C/06699938be04491a822b2e05301c55a8, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/20e761db5975fb9082f870bf056bd078/C/6c289f99aeaa4124bd3fd318af17610c, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/20e761db5975fb9082f870bf056bd078/C/138a4607009640ba9d4821f3f7e6e712, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/20e761db5975fb9082f870bf056bd078/C/3d39f7c1089b41b48ed37ca52be51bb3, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/20e761db5975fb9082f870bf056bd078/C/def7bce379f0445f9fdfd2b2f011416a, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/20e761db5975fb9082f870bf056bd078/C/633488e8312e4475b90ebf1ca8f775dd, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/20e761db5975fb9082f870bf056bd078/C/8a7b83ae8b904dcda979b2722064752f, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/20e761db5975fb9082f870bf056bd078/C/2157590b841f4c6da8be924763d5cf67, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/20e761db5975fb9082f870bf056bd078/C/50e1c4ae2a13474aa591f904b786e56e, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/20e761db5975fb9082f870bf056bd078/C/df0ecae983e54117bd14be37866047fe] to archive 2023-04-15 02:56:24,464 DEBUG [StoreCloser-TestAcidGuarantees,,1681527356526.20e761db5975fb9082f870bf056bd078.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-15 02:56:24,465 DEBUG [StoreCloser-TestAcidGuarantees,,1681527356526.20e761db5975fb9082f870bf056bd078.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/20e761db5975fb9082f870bf056bd078/C/bc0770e966664656b1a39dc2b7bc02ad to hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/archive/data/default/TestAcidGuarantees/20e761db5975fb9082f870bf056bd078/C/bc0770e966664656b1a39dc2b7bc02ad 2023-04-15 02:56:24,466 DEBUG [StoreCloser-TestAcidGuarantees,,1681527356526.20e761db5975fb9082f870bf056bd078.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/20e761db5975fb9082f870bf056bd078/C/3d55dce025b5415982c8518bb01bd7d4 to hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/archive/data/default/TestAcidGuarantees/20e761db5975fb9082f870bf056bd078/C/3d55dce025b5415982c8518bb01bd7d4 2023-04-15 02:56:24,467 DEBUG [StoreCloser-TestAcidGuarantees,,1681527356526.20e761db5975fb9082f870bf056bd078.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/20e761db5975fb9082f870bf056bd078/C/6dd346ee5c4e4a1f8bf4d4dff5b0287c to hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/archive/data/default/TestAcidGuarantees/20e761db5975fb9082f870bf056bd078/C/6dd346ee5c4e4a1f8bf4d4dff5b0287c 2023-04-15 02:56:24,468 DEBUG [StoreCloser-TestAcidGuarantees,,1681527356526.20e761db5975fb9082f870bf056bd078.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/20e761db5975fb9082f870bf056bd078/C/43db6e1add50473786a2b33dee3985cd to hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/archive/data/default/TestAcidGuarantees/20e761db5975fb9082f870bf056bd078/C/43db6e1add50473786a2b33dee3985cd 2023-04-15 02:56:24,469 DEBUG [StoreCloser-TestAcidGuarantees,,1681527356526.20e761db5975fb9082f870bf056bd078.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/20e761db5975fb9082f870bf056bd078/C/06699938be04491a822b2e05301c55a8 to hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/archive/data/default/TestAcidGuarantees/20e761db5975fb9082f870bf056bd078/C/06699938be04491a822b2e05301c55a8 2023-04-15 02:56:24,470 DEBUG [StoreCloser-TestAcidGuarantees,,1681527356526.20e761db5975fb9082f870bf056bd078.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/20e761db5975fb9082f870bf056bd078/C/6c289f99aeaa4124bd3fd318af17610c to hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/archive/data/default/TestAcidGuarantees/20e761db5975fb9082f870bf056bd078/C/6c289f99aeaa4124bd3fd318af17610c 2023-04-15 02:56:24,471 DEBUG [StoreCloser-TestAcidGuarantees,,1681527356526.20e761db5975fb9082f870bf056bd078.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/20e761db5975fb9082f870bf056bd078/C/138a4607009640ba9d4821f3f7e6e712 to hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/archive/data/default/TestAcidGuarantees/20e761db5975fb9082f870bf056bd078/C/138a4607009640ba9d4821f3f7e6e712 2023-04-15 02:56:24,472 DEBUG [StoreCloser-TestAcidGuarantees,,1681527356526.20e761db5975fb9082f870bf056bd078.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/20e761db5975fb9082f870bf056bd078/C/3d39f7c1089b41b48ed37ca52be51bb3 to hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/archive/data/default/TestAcidGuarantees/20e761db5975fb9082f870bf056bd078/C/3d39f7c1089b41b48ed37ca52be51bb3 2023-04-15 02:56:24,472 DEBUG [StoreCloser-TestAcidGuarantees,,1681527356526.20e761db5975fb9082f870bf056bd078.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/20e761db5975fb9082f870bf056bd078/C/def7bce379f0445f9fdfd2b2f011416a to hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/archive/data/default/TestAcidGuarantees/20e761db5975fb9082f870bf056bd078/C/def7bce379f0445f9fdfd2b2f011416a 2023-04-15 02:56:24,473 DEBUG [StoreCloser-TestAcidGuarantees,,1681527356526.20e761db5975fb9082f870bf056bd078.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/20e761db5975fb9082f870bf056bd078/C/633488e8312e4475b90ebf1ca8f775dd to hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/archive/data/default/TestAcidGuarantees/20e761db5975fb9082f870bf056bd078/C/633488e8312e4475b90ebf1ca8f775dd 2023-04-15 02:56:24,474 DEBUG [StoreCloser-TestAcidGuarantees,,1681527356526.20e761db5975fb9082f870bf056bd078.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/20e761db5975fb9082f870bf056bd078/C/8a7b83ae8b904dcda979b2722064752f to hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/archive/data/default/TestAcidGuarantees/20e761db5975fb9082f870bf056bd078/C/8a7b83ae8b904dcda979b2722064752f 2023-04-15 02:56:24,474 DEBUG [StoreCloser-TestAcidGuarantees,,1681527356526.20e761db5975fb9082f870bf056bd078.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/20e761db5975fb9082f870bf056bd078/C/2157590b841f4c6da8be924763d5cf67 to hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/archive/data/default/TestAcidGuarantees/20e761db5975fb9082f870bf056bd078/C/2157590b841f4c6da8be924763d5cf67 2023-04-15 02:56:24,475 DEBUG [StoreCloser-TestAcidGuarantees,,1681527356526.20e761db5975fb9082f870bf056bd078.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/20e761db5975fb9082f870bf056bd078/C/50e1c4ae2a13474aa591f904b786e56e to hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/archive/data/default/TestAcidGuarantees/20e761db5975fb9082f870bf056bd078/C/50e1c4ae2a13474aa591f904b786e56e 2023-04-15 02:56:24,476 DEBUG [StoreCloser-TestAcidGuarantees,,1681527356526.20e761db5975fb9082f870bf056bd078.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/20e761db5975fb9082f870bf056bd078/C/df0ecae983e54117bd14be37866047fe to hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/archive/data/default/TestAcidGuarantees/20e761db5975fb9082f870bf056bd078/C/df0ecae983e54117bd14be37866047fe 2023-04-15 02:56:24,479 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/default/TestAcidGuarantees/20e761db5975fb9082f870bf056bd078/recovered.edits/412.seqid, newMaxSeqId=412, maxSeqId=4 2023-04-15 02:56:24,480 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1681527356526.20e761db5975fb9082f870bf056bd078. 2023-04-15 02:56:24,480 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1558): Region close journal for 20e761db5975fb9082f870bf056bd078: 2023-04-15 02:56:24,481 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] handler.UnassignRegionHandler(147): Closed 20e761db5975fb9082f870bf056bd078 2023-04-15 02:56:24,482 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=63 updating hbase:meta row=20e761db5975fb9082f870bf056bd078, regionState=CLOSED 2023-04-15 02:56:24,482 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1681527356526.20e761db5975fb9082f870bf056bd078.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1681527384482"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1681527384482"}]},"ts":"1681527384482"} 2023-04-15 02:56:24,484 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=64, resume processing ppid=63 2023-04-15 02:56:24,484 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=64, ppid=63, state=SUCCESS; CloseRegionProcedure 20e761db5975fb9082f870bf056bd078, server=jenkins-hbase20.apache.org,34681,1681527209133 in 2.7660 sec 2023-04-15 02:56:24,485 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=63, resume processing ppid=62 2023-04-15 02:56:24,485 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=63, ppid=62, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=20e761db5975fb9082f870bf056bd078, UNASSIGN in 2.7710 sec 2023-04-15 02:56:24,485 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1681527384485"}]},"ts":"1681527384485"} 2023-04-15 02:56:24,485 INFO [PEWorker-2] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=DISABLED in hbase:meta 2023-04-15 02:56:24,486 INFO [PEWorker-2] procedure.DisableTableProcedure(305): Set TestAcidGuarantees to state=DISABLED 2023-04-15 02:56:24,487 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=62, state=SUCCESS; DisableTableProcedure table=TestAcidGuarantees in 2.7880 sec 2023-04-15 02:56:25,812 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(1227): Checking to see if procedure is done pid=62 2023-04-15 02:56:25,813 INFO [Listener at localhost.localdomain/33693] client.HBaseAdmin$TableFuture(3541): Operation: DISABLE, Table Name: default:TestAcidGuarantees, procId: 62 completed 2023-04-15 02:56:25,815 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.HMaster$5(2228): Client=jenkins//148.251.75.209 delete TestAcidGuarantees 2023-04-15 02:56:25,816 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] procedure2.ProcedureExecutor(1029): Stored pid=65, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION; DeleteTableProcedure table=TestAcidGuarantees 2023-04-15 02:56:25,819 DEBUG [PEWorker-4] procedure.DeleteTableProcedure(101): Waiting for RIT for pid=65, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-15 02:56:25,821 DEBUG [PEWorker-4] procedure.DeleteTableProcedure(113): Deleting regions from filesystem for pid=65, state=RUNNABLE:DELETE_TABLE_CLEAR_FS_LAYOUT, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-15 02:56:25,821 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(1227): Checking to see if procedure is done pid=65 2023-04-15 02:56:25,827 DEBUG [HFileArchiver-13] backup.HFileArchiver(131): ARCHIVING hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/.tmp/data/default/TestAcidGuarantees/20e761db5975fb9082f870bf056bd078 2023-04-15 02:56:25,830 DEBUG [HFileArchiver-13] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/.tmp/data/default/TestAcidGuarantees/20e761db5975fb9082f870bf056bd078/A, FileablePath, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/.tmp/data/default/TestAcidGuarantees/20e761db5975fb9082f870bf056bd078/B, FileablePath, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/.tmp/data/default/TestAcidGuarantees/20e761db5975fb9082f870bf056bd078/C, FileablePath, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/.tmp/data/default/TestAcidGuarantees/20e761db5975fb9082f870bf056bd078/recovered.edits] 2023-04-15 02:56:25,833 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/.tmp/data/default/TestAcidGuarantees/20e761db5975fb9082f870bf056bd078/A/44475d30540b4839a6d78e65f3b7ce28 to hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/archive/data/default/TestAcidGuarantees/20e761db5975fb9082f870bf056bd078/A/44475d30540b4839a6d78e65f3b7ce28 2023-04-15 02:56:25,835 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/.tmp/data/default/TestAcidGuarantees/20e761db5975fb9082f870bf056bd078/A/f8cd1ab909254accb88a7e0be3ab020b to hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/archive/data/default/TestAcidGuarantees/20e761db5975fb9082f870bf056bd078/A/f8cd1ab909254accb88a7e0be3ab020b 2023-04-15 02:56:25,838 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/.tmp/data/default/TestAcidGuarantees/20e761db5975fb9082f870bf056bd078/B/bad5e513a0fe40ae84e0f11a25ead334 to hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/archive/data/default/TestAcidGuarantees/20e761db5975fb9082f870bf056bd078/B/bad5e513a0fe40ae84e0f11a25ead334 2023-04-15 02:56:25,839 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/.tmp/data/default/TestAcidGuarantees/20e761db5975fb9082f870bf056bd078/B/d6057730ef64497da7dda5ab90dc2c30 to hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/archive/data/default/TestAcidGuarantees/20e761db5975fb9082f870bf056bd078/B/d6057730ef64497da7dda5ab90dc2c30 2023-04-15 02:56:25,842 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/.tmp/data/default/TestAcidGuarantees/20e761db5975fb9082f870bf056bd078/C/2af9856544734ee3a41835a2bcc002e1 to hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/archive/data/default/TestAcidGuarantees/20e761db5975fb9082f870bf056bd078/C/2af9856544734ee3a41835a2bcc002e1 2023-04-15 02:56:25,844 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/.tmp/data/default/TestAcidGuarantees/20e761db5975fb9082f870bf056bd078/C/3db79db9269a42dfaf6d4d15d1e01dea to hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/archive/data/default/TestAcidGuarantees/20e761db5975fb9082f870bf056bd078/C/3db79db9269a42dfaf6d4d15d1e01dea 2023-04-15 02:56:25,845 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/.tmp/data/default/TestAcidGuarantees/20e761db5975fb9082f870bf056bd078/C/5a7f92071f5645c08e36bdeaaf9bcf47 to hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/archive/data/default/TestAcidGuarantees/20e761db5975fb9082f870bf056bd078/C/5a7f92071f5645c08e36bdeaaf9bcf47 2023-04-15 02:56:25,847 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/.tmp/data/default/TestAcidGuarantees/20e761db5975fb9082f870bf056bd078/C/7e2ca3a28dc249e9a1d84c90efb5603f to hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/archive/data/default/TestAcidGuarantees/20e761db5975fb9082f870bf056bd078/C/7e2ca3a28dc249e9a1d84c90efb5603f 2023-04-15 02:56:25,849 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/.tmp/data/default/TestAcidGuarantees/20e761db5975fb9082f870bf056bd078/C/8828de42aa92431d8eddbf66abcc5537 to hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/archive/data/default/TestAcidGuarantees/20e761db5975fb9082f870bf056bd078/C/8828de42aa92431d8eddbf66abcc5537 2023-04-15 02:56:25,851 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/.tmp/data/default/TestAcidGuarantees/20e761db5975fb9082f870bf056bd078/C/9f88097737a545c5bc685f2c98f3b22e to hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/archive/data/default/TestAcidGuarantees/20e761db5975fb9082f870bf056bd078/C/9f88097737a545c5bc685f2c98f3b22e 2023-04-15 02:56:25,853 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/.tmp/data/default/TestAcidGuarantees/20e761db5975fb9082f870bf056bd078/C/bc23e3640fa7498094a8d56ae64ac1a1 to hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/archive/data/default/TestAcidGuarantees/20e761db5975fb9082f870bf056bd078/C/bc23e3640fa7498094a8d56ae64ac1a1 2023-04-15 02:56:25,854 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/.tmp/data/default/TestAcidGuarantees/20e761db5975fb9082f870bf056bd078/C/d230775c80b84829871afc8a529d4cf9 to hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/archive/data/default/TestAcidGuarantees/20e761db5975fb9082f870bf056bd078/C/d230775c80b84829871afc8a529d4cf9 2023-04-15 02:56:25,858 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/.tmp/data/default/TestAcidGuarantees/20e761db5975fb9082f870bf056bd078/recovered.edits/412.seqid to hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/archive/data/default/TestAcidGuarantees/20e761db5975fb9082f870bf056bd078/recovered.edits/412.seqid 2023-04-15 02:56:25,859 DEBUG [HFileArchiver-13] backup.HFileArchiver(596): Deleted hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/.tmp/data/default/TestAcidGuarantees/20e761db5975fb9082f870bf056bd078 2023-04-15 02:56:25,859 DEBUG [PEWorker-4] procedure.DeleteTableProcedure(328): Archived TestAcidGuarantees regions 2023-04-15 02:56:25,859 DEBUG [PEWorker-4] backup.HFileArchiver(131): ARCHIVING hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3 2023-04-15 02:56:25,860 DEBUG [PEWorker-4] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A] 2023-04-15 02:56:25,864 DEBUG [PEWorker-4] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023041507c813b405274e1e90689e5288332391 to hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023041507c813b405274e1e90689e5288332391 2023-04-15 02:56:25,866 DEBUG [PEWorker-4] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023041518ea87b6f4df457d9bda1542999cb119 to hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023041518ea87b6f4df457d9bda1542999cb119 2023-04-15 02:56:25,867 DEBUG [PEWorker-4] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023041523555c1aa3954069828223303aece98d to hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023041523555c1aa3954069828223303aece98d 2023-04-15 02:56:25,868 DEBUG [PEWorker-4] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023041537ea055f08164e7fab9c6d42d004c4ab to hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023041537ea055f08164e7fab9c6d42d004c4ab 2023-04-15 02:56:25,869 DEBUG [PEWorker-4] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230415614efc38fedf4a8ca58f0196c5d5df6b to hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230415614efc38fedf4a8ca58f0196c5d5df6b 2023-04-15 02:56:25,870 DEBUG [PEWorker-4] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023041570800ab6e65541ff9ec74ec1decb3627 to hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023041570800ab6e65541ff9ec74ec1decb3627 2023-04-15 02:56:25,872 DEBUG [PEWorker-4] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230415789f7fdd58984ddba60fade421b9c646 to hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230415789f7fdd58984ddba60fade421b9c646 2023-04-15 02:56:25,873 DEBUG [PEWorker-4] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304157b3b2b4285f7431497122d84df30463c to hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304157b3b2b4285f7431497122d84df30463c 2023-04-15 02:56:25,874 DEBUG [PEWorker-4] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023041581b96046be2248c9a6096a74bbe1191c to hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023041581b96046be2248c9a6096a74bbe1191c 2023-04-15 02:56:25,876 DEBUG [PEWorker-4] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230415a7ba445cc5ae4e4cbfb7f9025e44832d to hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230415a7ba445cc5ae4e4cbfb7f9025e44832d 2023-04-15 02:56:25,877 DEBUG [PEWorker-4] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230415b0f6cf38abe84579bf13c51e746e826c to hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230415b0f6cf38abe84579bf13c51e746e826c 2023-04-15 02:56:25,878 DEBUG [PEWorker-4] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230415b159c1baef0e41639ac587235c017ee1 to hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230415b159c1baef0e41639ac587235c017ee1 2023-04-15 02:56:25,880 DEBUG [PEWorker-4] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230415b77ef5b7fa79425abda9841fea27d12c to hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230415b77ef5b7fa79425abda9841fea27d12c 2023-04-15 02:56:25,882 DEBUG [PEWorker-4] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230415bccd2248d1f64c7ba7ae153b79a138f0 to hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230415bccd2248d1f64c7ba7ae153b79a138f0 2023-04-15 02:56:25,883 DEBUG [PEWorker-4] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230415be78ae3e463e48a2862ae704a1c157c3 to hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230415be78ae3e463e48a2862ae704a1c157c3 2023-04-15 02:56:25,885 DEBUG [PEWorker-4] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230415d125e115f94e44489764a75d1d59518d to hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230415d125e115f94e44489764a75d1d59518d 2023-04-15 02:56:25,886 DEBUG [PEWorker-4] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230415d8f2a6ce669d4ed88252e79e7cb275a4 to hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230415d8f2a6ce669d4ed88252e79e7cb275a4 2023-04-15 02:56:25,888 DEBUG [PEWorker-4] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230415f96606d3392c4f5cb5058787f9e075d6 to hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230415f96606d3392c4f5cb5058787f9e075d6 2023-04-15 02:56:25,889 DEBUG [PEWorker-4] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230415fd633ad9c04941d1a47e1d7f15f6d9a6 to hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230415fd633ad9c04941d1a47e1d7f15f6d9a6 2023-04-15 02:56:25,890 DEBUG [PEWorker-4] backup.HFileArchiver(596): Deleted hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3 2023-04-15 02:56:25,892 DEBUG [PEWorker-4] procedure.DeleteTableProcedure(118): Deleting regions from META for pid=65, state=RUNNABLE:DELETE_TABLE_REMOVE_FROM_META, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-15 02:56:25,894 WARN [PEWorker-4] procedure.DeleteTableProcedure(384): Deleting some vestigial 1 rows of TestAcidGuarantees from hbase:meta 2023-04-15 02:56:25,896 DEBUG [PEWorker-4] procedure.DeleteTableProcedure(421): Removing 'TestAcidGuarantees' descriptor. 2023-04-15 02:56:25,897 DEBUG [PEWorker-4] procedure.DeleteTableProcedure(124): Deleting assignment state for pid=65, state=RUNNABLE:DELETE_TABLE_UNASSIGN_REGIONS, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-15 02:56:25,897 DEBUG [PEWorker-4] procedure.DeleteTableProcedure(411): Removing 'TestAcidGuarantees' from region states. 2023-04-15 02:56:25,897 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"TestAcidGuarantees,,1681527356526.20e761db5975fb9082f870bf056bd078.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1681527385897"}]},"ts":"9223372036854775807"} 2023-04-15 02:56:25,899 INFO [PEWorker-4] hbase.MetaTableAccessor(1788): Deleted 1 regions from META 2023-04-15 02:56:25,899 DEBUG [PEWorker-4] hbase.MetaTableAccessor(1789): Deleted regions: [{ENCODED => 20e761db5975fb9082f870bf056bd078, NAME => 'TestAcidGuarantees,,1681527356526.20e761db5975fb9082f870bf056bd078.', STARTKEY => '', ENDKEY => ''}] 2023-04-15 02:56:25,899 DEBUG [PEWorker-4] procedure.DeleteTableProcedure(415): Marking 'TestAcidGuarantees' as deleted. 2023-04-15 02:56:25,899 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":0,"tag":[],"timestamp":"1681527385899"}]},"ts":"9223372036854775807"} 2023-04-15 02:56:25,901 INFO [PEWorker-4] hbase.MetaTableAccessor(1658): Deleted table TestAcidGuarantees state from META 2023-04-15 02:56:25,902 DEBUG [PEWorker-4] procedure.DeleteTableProcedure(130): Finished pid=65, state=RUNNABLE:DELETE_TABLE_POST_OPERATION, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-15 02:56:25,903 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=65, state=SUCCESS; DeleteTableProcedure table=TestAcidGuarantees in 87 msec 2023-04-15 02:56:25,922 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43619] master.MasterRpcServices(1227): Checking to see if procedure is done pid=65 2023-04-15 02:56:25,923 INFO [Listener at localhost.localdomain/33693] client.HBaseAdmin$TableFuture(3541): Operation: DELETE, Table Name: default:TestAcidGuarantees, procId: 65 completed 2023-04-15 02:56:25,940 INFO [Listener at localhost.localdomain/33693] hbase.ResourceChecker(175): after: TestAcidGuaranteesWithEagerPolicy#testMobGetAtomicity Thread=227 (was 230), OpenFileDescriptor=532 (was 537), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=297 (was 362), ProcessCount=168 (was 168), AvailableMemoryMB=5766 (was 5681) - AvailableMemoryMB LEAK? - 2023-04-15 02:56:25,941 INFO [Listener at localhost.localdomain/33693] hbase.HBaseTestingUtility(1286): Shutting down minicluster 2023-04-15 02:56:25,941 INFO [Listener at localhost.localdomain/33693] client.ConnectionImplementation(1974): Closing master protocol: MasterService 2023-04-15 02:56:25,941 DEBUG [Listener at localhost.localdomain/33693] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x056ecc29 to 127.0.0.1:51762 2023-04-15 02:56:25,941 DEBUG [Listener at localhost.localdomain/33693] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-15 02:56:25,942 DEBUG [Listener at localhost.localdomain/33693] util.JVMClusterUtil(237): Shutting down HBase Cluster 2023-04-15 02:56:25,942 DEBUG [Listener at localhost.localdomain/33693] util.JVMClusterUtil(257): Found active master hash=1290719582, stopped=false 2023-04-15 02:56:25,942 INFO [Listener at localhost.localdomain/33693] master.ServerManager(901): Cluster shutdown requested of master=jenkins-hbase20.apache.org,43619,1681527208212 2023-04-15 02:56:25,943 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/running 2023-04-15 02:56:25,943 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/running 2023-04-15 02:56:25,943 INFO [Listener at localhost.localdomain/33693] procedure2.ProcedureExecutor(629): Stopping 2023-04-15 02:56:25,943 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-04-15 02:56:25,943 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2023-04-15 02:56:25,943 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2023-04-15 02:56:25,943 DEBUG [Listener at localhost.localdomain/33693] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x2a875802 to 127.0.0.1:51762 2023-04-15 02:56:25,944 DEBUG [Listener at localhost.localdomain/33693] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-15 02:56:25,944 INFO [Listener at localhost.localdomain/33693] regionserver.HRegionServer(2296): ***** STOPPING region server 'jenkins-hbase20.apache.org,34681,1681527209133' ***** 2023-04-15 02:56:25,944 INFO [Listener at localhost.localdomain/33693] regionserver.HRegionServer(2310): STOPPED: Shutdown requested 2023-04-15 02:56:25,944 INFO [RS:0;jenkins-hbase20:34681] regionserver.HeapMemoryManager(220): Stopping 2023-04-15 02:56:25,944 INFO [RS:0;jenkins-hbase20:34681] flush.RegionServerFlushTableProcedureManager(117): Stopping region server flush procedure manager gracefully. 2023-04-15 02:56:25,944 INFO [MemStoreFlusher.0] regionserver.MemStoreFlusher$FlushHandler(361): MemStoreFlusher.0 exiting 2023-04-15 02:56:25,944 INFO [RS:0;jenkins-hbase20:34681] snapshot.RegionServerSnapshotManager(137): Stopping RegionServerSnapshotManager gracefully. 2023-04-15 02:56:25,945 INFO [RS:0;jenkins-hbase20:34681] regionserver.HRegionServer(3304): Received CLOSE for 5505718cae8473d3311d920897677a71 2023-04-15 02:56:25,945 INFO [RS:0;jenkins-hbase20:34681] regionserver.HRegionServer(1145): stopping server jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:25,946 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1604): Closing 5505718cae8473d3311d920897677a71, disabling compactions & flushes 2023-04-15 02:56:25,946 DEBUG [RS:0;jenkins-hbase20:34681] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x7ac38bcd to 127.0.0.1:51762 2023-04-15 02:56:25,946 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1626): Closing region hbase:namespace,,1681527211546.5505718cae8473d3311d920897677a71. 2023-04-15 02:56:25,946 DEBUG [RS:0;jenkins-hbase20:34681] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-15 02:56:25,946 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on hbase:namespace,,1681527211546.5505718cae8473d3311d920897677a71. 2023-04-15 02:56:25,946 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1714): Acquired close lock on hbase:namespace,,1681527211546.5505718cae8473d3311d920897677a71. after waiting 0 ms 2023-04-15 02:56:25,946 INFO [RS:0;jenkins-hbase20:34681] regionserver.CompactSplit(434): Waiting for Split Thread to finish... 2023-04-15 02:56:25,946 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1724): Updates disabled for region hbase:namespace,,1681527211546.5505718cae8473d3311d920897677a71. 2023-04-15 02:56:25,946 INFO [RS:0;jenkins-hbase20:34681] regionserver.CompactSplit(434): Waiting for Large Compaction Thread to finish... 2023-04-15 02:56:25,946 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(2745): Flushing 5505718cae8473d3311d920897677a71 1/1 column families, dataSize=78 B heapSize=488 B 2023-04-15 02:56:25,946 INFO [RS:0;jenkins-hbase20:34681] regionserver.CompactSplit(434): Waiting for Small Compaction Thread to finish... 2023-04-15 02:56:25,946 INFO [RS:0;jenkins-hbase20:34681] regionserver.HRegionServer(3304): Received CLOSE for 1588230740 2023-04-15 02:56:25,946 INFO [RS:0;jenkins-hbase20:34681] regionserver.HRegionServer(1475): Waiting on 2 regions to close 2023-04-15 02:56:25,947 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1604): Closing 1588230740, disabling compactions & flushes 2023-04-15 02:56:25,947 DEBUG [RS:0;jenkins-hbase20:34681] regionserver.HRegionServer(1479): Online Regions={5505718cae8473d3311d920897677a71=hbase:namespace,,1681527211546.5505718cae8473d3311d920897677a71., 1588230740=hbase:meta,,1.1588230740} 2023-04-15 02:56:25,947 INFO [RS_CLOSE_META-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1626): Closing region hbase:meta,,1.1588230740 2023-04-15 02:56:25,947 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on hbase:meta,,1.1588230740 2023-04-15 02:56:25,947 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1714): Acquired close lock on hbase:meta,,1.1588230740 after waiting 0 ms 2023-04-15 02:56:25,947 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1724): Updates disabled for region hbase:meta,,1.1588230740 2023-04-15 02:56:25,947 INFO [RS_CLOSE_META-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(2745): Flushing 1588230740 3/3 column families, dataSize=20.94 KB heapSize=36.31 KB 2023-04-15 02:56:25,949 DEBUG [RS:0;jenkins-hbase20:34681] regionserver.HRegionServer(1505): Waiting on 1588230740, 5505718cae8473d3311d920897677a71 2023-04-15 02:56:26,019 INFO [regionserver/jenkins-hbase20:0.leaseChecker] regionserver.LeaseManager(133): Closed leases 2023-04-15 02:56:26,149 DEBUG [RS:0;jenkins-hbase20:34681] regionserver.HRegionServer(1505): Waiting on 1588230740, 5505718cae8473d3311d920897677a71 2023-04-15 02:56:26,162 INFO [regionserver/jenkins-hbase20:0.Chore.1] hbase.ScheduledChore(146): Chore: CompactionChecker was stopped 2023-04-15 02:56:26,162 INFO [regionserver/jenkins-hbase20:0.Chore.1] hbase.ScheduledChore(146): Chore: MemstoreFlusherChore was stopped 2023-04-15 02:56:26,349 DEBUG [RS:0;jenkins-hbase20:34681] regionserver.HRegionServer(1505): Waiting on 1588230740, 5505718cae8473d3311d920897677a71 2023-04-15 02:56:26,357 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=78 B at sequenceid=6 (bloomFilter=true), to=hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/hbase/namespace/5505718cae8473d3311d920897677a71/.tmp/info/d05f6323adec4ae180d5b65642095303 2023-04-15 02:56:26,357 INFO [RS_CLOSE_META-regionserver/jenkins-hbase20:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=18.29 KB at sequenceid=93 (bloomFilter=false), to=hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/hbase/meta/1588230740/.tmp/info/3f92e10c2493459595bedef316535db8 2023-04-15 02:56:26,366 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/hbase/namespace/5505718cae8473d3311d920897677a71/.tmp/info/d05f6323adec4ae180d5b65642095303 as hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/hbase/namespace/5505718cae8473d3311d920897677a71/info/d05f6323adec4ae180d5b65642095303 2023-04-15 02:56:26,367 INFO [RS_CLOSE_META-regionserver/jenkins-hbase20:0-0] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for 3f92e10c2493459595bedef316535db8 2023-04-15 02:56:26,375 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/hbase/namespace/5505718cae8473d3311d920897677a71/info/d05f6323adec4ae180d5b65642095303, entries=2, sequenceid=6, filesize=4.8 K 2023-04-15 02:56:26,375 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(2948): Finished flush of dataSize ~78 B/78, heapSize ~472 B/472, currentSize=0 B/0 for 5505718cae8473d3311d920897677a71 in 429ms, sequenceid=6, compaction requested=false 2023-04-15 02:56:26,381 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/hbase/namespace/5505718cae8473d3311d920897677a71/recovered.edits/9.seqid, newMaxSeqId=9, maxSeqId=1 2023-04-15 02:56:26,382 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1838): Closed hbase:namespace,,1681527211546.5505718cae8473d3311d920897677a71. 2023-04-15 02:56:26,382 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1558): Region close journal for 5505718cae8473d3311d920897677a71: 2023-04-15 02:56:26,382 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] handler.CloseRegionHandler(117): Closed hbase:namespace,,1681527211546.5505718cae8473d3311d920897677a71. 2023-04-15 02:56:26,550 DEBUG [RS:0;jenkins-hbase20:34681] regionserver.HRegionServer(1505): Waiting on 1588230740 2023-04-15 02:56:26,750 DEBUG [RS:0;jenkins-hbase20:34681] regionserver.HRegionServer(1505): Waiting on 1588230740 2023-04-15 02:56:26,780 INFO [RS_CLOSE_META-regionserver/jenkins-hbase20:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=588 B at sequenceid=93 (bloomFilter=false), to=hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/hbase/meta/1588230740/.tmp/rep_barrier/16389c2e36324046a3a4c51b937748fa 2023-04-15 02:56:26,788 INFO [RS_CLOSE_META-regionserver/jenkins-hbase20:0-0] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for 16389c2e36324046a3a4c51b937748fa 2023-04-15 02:56:26,950 INFO [RS:0;jenkins-hbase20:34681] regionserver.HRegionServer(1475): Waiting on 1 regions to close 2023-04-15 02:56:26,951 DEBUG [RS:0;jenkins-hbase20:34681] regionserver.HRegionServer(1479): Online Regions={1588230740=hbase:meta,,1.1588230740} 2023-04-15 02:56:26,951 DEBUG [RS:0;jenkins-hbase20:34681] regionserver.HRegionServer(1505): Waiting on 1588230740 2023-04-15 02:56:27,151 DEBUG [RS:0;jenkins-hbase20:34681] regionserver.HRegionServer(1505): Waiting on 1588230740 2023-04-15 02:56:27,197 INFO [RS_CLOSE_META-regionserver/jenkins-hbase20:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=2.08 KB at sequenceid=93 (bloomFilter=false), to=hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/hbase/meta/1588230740/.tmp/table/9e4659a261334e8c92709ccbf8f30cd2 2023-04-15 02:56:27,201 INFO [RS_CLOSE_META-regionserver/jenkins-hbase20:0-0] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for 9e4659a261334e8c92709ccbf8f30cd2 2023-04-15 02:56:27,202 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase20:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/hbase/meta/1588230740/.tmp/info/3f92e10c2493459595bedef316535db8 as hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/hbase/meta/1588230740/info/3f92e10c2493459595bedef316535db8 2023-04-15 02:56:27,206 INFO [RS_CLOSE_META-regionserver/jenkins-hbase20:0-0] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for 3f92e10c2493459595bedef316535db8 2023-04-15 02:56:27,206 INFO [RS_CLOSE_META-regionserver/jenkins-hbase20:0-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/hbase/meta/1588230740/info/3f92e10c2493459595bedef316535db8, entries=22, sequenceid=93, filesize=7.1 K 2023-04-15 02:56:27,208 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase20:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/hbase/meta/1588230740/.tmp/rep_barrier/16389c2e36324046a3a4c51b937748fa as hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/hbase/meta/1588230740/rep_barrier/16389c2e36324046a3a4c51b937748fa 2023-04-15 02:56:27,213 INFO [RS_CLOSE_META-regionserver/jenkins-hbase20:0-0] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for 16389c2e36324046a3a4c51b937748fa 2023-04-15 02:56:27,213 INFO [RS_CLOSE_META-regionserver/jenkins-hbase20:0-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/hbase/meta/1588230740/rep_barrier/16389c2e36324046a3a4c51b937748fa, entries=6, sequenceid=93, filesize=5.5 K 2023-04-15 02:56:27,214 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase20:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/hbase/meta/1588230740/.tmp/table/9e4659a261334e8c92709ccbf8f30cd2 as hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/hbase/meta/1588230740/table/9e4659a261334e8c92709ccbf8f30cd2 2023-04-15 02:56:27,219 INFO [RS_CLOSE_META-regionserver/jenkins-hbase20:0-0] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for 9e4659a261334e8c92709ccbf8f30cd2 2023-04-15 02:56:27,220 INFO [RS_CLOSE_META-regionserver/jenkins-hbase20:0-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/hbase/meta/1588230740/table/9e4659a261334e8c92709ccbf8f30cd2, entries=9, sequenceid=93, filesize=5.5 K 2023-04-15 02:56:27,220 INFO [RS_CLOSE_META-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(2948): Finished flush of dataSize ~20.94 KB/21446, heapSize ~36.27 KB/37136, currentSize=0 B/0 for 1588230740 in 1273ms, sequenceid=93, compaction requested=false 2023-04-15 02:56:27,232 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase20:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/data/hbase/meta/1588230740/recovered.edits/96.seqid, newMaxSeqId=96, maxSeqId=1 2023-04-15 02:56:27,233 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase20:0-0] coprocessor.CoprocessorHost(310): Stop coprocessor org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint 2023-04-15 02:56:27,234 INFO [RS_CLOSE_META-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1838): Closed hbase:meta,,1.1588230740 2023-04-15 02:56:27,234 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1558): Region close journal for 1588230740: 2023-04-15 02:56:27,234 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase20:0-0] handler.CloseRegionHandler(117): Closed hbase:meta,,1.1588230740 2023-04-15 02:56:27,351 INFO [RS:0;jenkins-hbase20:34681] regionserver.HRegionServer(1171): stopping server jenkins-hbase20.apache.org,34681,1681527209133; all regions closed. 2023-04-15 02:56:27,364 DEBUG [RS:0;jenkins-hbase20:34681] wal.AbstractFSWAL(1028): Moved 1 WAL file(s) to /user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/oldWALs 2023-04-15 02:56:27,364 INFO [RS:0;jenkins-hbase20:34681] wal.AbstractFSWAL(1031): Closed WAL: AsyncFSWAL jenkins-hbase20.apache.org%2C34681%2C1681527209133.meta:.meta(num 1681527211312) 2023-04-15 02:56:27,369 DEBUG [RS:0;jenkins-hbase20:34681] wal.AbstractFSWAL(1028): Moved 1 WAL file(s) to /user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/oldWALs 2023-04-15 02:56:27,369 INFO [RS:0;jenkins-hbase20:34681] wal.AbstractFSWAL(1031): Closed WAL: AsyncFSWAL jenkins-hbase20.apache.org%2C34681%2C1681527209133:(num 1681527210309) 2023-04-15 02:56:27,369 DEBUG [RS:0;jenkins-hbase20:34681] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-15 02:56:27,369 INFO [RS:0;jenkins-hbase20:34681] regionserver.LeaseManager(133): Closed leases 2023-04-15 02:56:27,369 INFO [RS:0;jenkins-hbase20:34681] hbase.ChoreService(369): Chore service for: regionserver/jenkins-hbase20:0 had [ScheduledChore name=CompactedHFilesCleaner, period=120000, unit=MILLISECONDS, ScheduledChore name=CompactionThroughputTuner, period=60000, unit=MILLISECONDS] on shutdown 2023-04-15 02:56:27,369 INFO [regionserver/jenkins-hbase20:0.logRoller] wal.AbstractWALRoller(243): LogRoller exiting. 2023-04-15 02:56:27,370 INFO [RS:0;jenkins-hbase20:34681] ipc.NettyRpcServer(158): Stopping server on /148.251.75.209:34681 2023-04-15 02:56:27,374 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-04-15 02:56:27,374 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/rs/jenkins-hbase20.apache.org,34681,1681527209133 2023-04-15 02:56:27,374 ERROR [Listener at localhost.localdomain/33693-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@45b8d741 rejected from java.util.concurrent.ThreadPoolExecutor@41874b2e[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 2335] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-04-15 02:56:27,375 INFO [RegionServerTracker-0] master.RegionServerTracker(179): RegionServer ephemeral node deleted, processing expiration [jenkins-hbase20.apache.org,34681,1681527209133] 2023-04-15 02:56:27,375 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-04-15 02:56:27,376 ERROR [Listener at localhost.localdomain/33693-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@364a6064 rejected from java.util.concurrent.ThreadPoolExecutor@41874b2e[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 2335] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-04-15 02:56:27,376 DEBUG [RegionServerTracker-0] master.DeadServer(103): Processing jenkins-hbase20.apache.org,34681,1681527209133; numProcessing=1 2023-04-15 02:56:27,377 DEBUG [RegionServerTracker-0] zookeeper.RecoverableZooKeeper(172): Node /hbase/draining/jenkins-hbase20.apache.org,34681,1681527209133 already deleted, retry=false 2023-04-15 02:56:27,377 INFO [RegionServerTracker-0] master.ServerManager(561): Cluster shutdown set; jenkins-hbase20.apache.org,34681,1681527209133 expired; onlineServers=0 2023-04-15 02:56:27,377 INFO [RegionServerTracker-0] regionserver.HRegionServer(2296): ***** STOPPING region server 'jenkins-hbase20.apache.org,43619,1681527208212' ***** 2023-04-15 02:56:27,377 INFO [RegionServerTracker-0] regionserver.HRegionServer(2310): STOPPED: Cluster shutdown set; onlineServer=0 2023-04-15 02:56:27,377 DEBUG [M:0;jenkins-hbase20:43619] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@dd3db65, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=jenkins-hbase20.apache.org/148.251.75.209:0 2023-04-15 02:56:27,377 INFO [M:0;jenkins-hbase20:43619] regionserver.HRegionServer(1145): stopping server jenkins-hbase20.apache.org,43619,1681527208212 2023-04-15 02:56:27,377 INFO [M:0;jenkins-hbase20:43619] regionserver.HRegionServer(1171): stopping server jenkins-hbase20.apache.org,43619,1681527208212; all regions closed. 2023-04-15 02:56:27,377 DEBUG [M:0;jenkins-hbase20:43619] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-15 02:56:27,378 DEBUG [M:0;jenkins-hbase20:43619] cleaner.LogCleaner(198): Cancelling LogCleaner 2023-04-15 02:56:27,378 WARN [OldWALsCleaner-0] cleaner.LogCleaner(186): Interrupted while cleaning old WALs, will try to clean it next round. Exiting. 2023-04-15 02:56:27,378 DEBUG [M:0;jenkins-hbase20:43619] cleaner.HFileCleaner(317): Stopping file delete threads 2023-04-15 02:56:27,378 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster-HFileCleaner.small.0-1681527210031] cleaner.HFileCleaner(288): Exit Thread[master/jenkins-hbase20:0:becomeActiveMaster-HFileCleaner.small.0-1681527210031,5,FailOnTimeoutGroup] 2023-04-15 02:56:27,378 INFO [M:0;jenkins-hbase20:43619] master.MasterMobCompactionThread(168): Waiting for Mob Compaction Thread to finish... 2023-04-15 02:56:27,378 INFO [M:0;jenkins-hbase20:43619] master.MasterMobCompactionThread(168): Waiting for Region Server Mob Compaction Thread to finish... 2023-04-15 02:56:27,378 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster-HFileCleaner.large.0-1681527210030] cleaner.HFileCleaner(288): Exit Thread[master/jenkins-hbase20:0:becomeActiveMaster-HFileCleaner.large.0-1681527210030,5,FailOnTimeoutGroup] 2023-04-15 02:56:27,378 INFO [M:0;jenkins-hbase20:43619] hbase.ChoreService(369): Chore service for: master/jenkins-hbase20:0 had [] on shutdown 2023-04-15 02:56:27,378 DEBUG [M:0;jenkins-hbase20:43619] master.HMaster(1512): Stopping service threads 2023-04-15 02:56:27,378 INFO [M:0;jenkins-hbase20:43619] procedure2.RemoteProcedureDispatcher(118): Stopping procedure remote dispatcher 2023-04-15 02:56:27,379 ERROR [M:0;jenkins-hbase20:43619] procedure2.ProcedureExecutor(653): ThreadGroup java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] contains running threads; null: See STDOUT java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] Thread[IPC Client (774823702) connection to localhost.localdomain/127.0.0.1:34069 from jenkins,5,PEWorkerGroup] Thread[HFileArchiver-13,5,PEWorkerGroup] 2023-04-15 02:56:27,379 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/master 2023-04-15 02:56:27,379 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-04-15 02:56:27,379 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2023-04-15 02:56:27,379 INFO [M:0;jenkins-hbase20:43619] region.RegionProcedureStore(113): Stopping the Region Procedure Store, isAbort=false 2023-04-15 02:56:27,379 DEBUG [normalizer-worker-0] normalizer.RegionNormalizerWorker(174): interrupt detected. terminating. 2023-04-15 02:56:27,380 DEBUG [M:0;jenkins-hbase20:43619] zookeeper.ZKUtil(398): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Unable to get data of znode /hbase/master because node does not exist (not an error) 2023-04-15 02:56:27,380 WARN [M:0;jenkins-hbase20:43619] master.ActiveMasterManager(326): Failed get of master address: java.io.IOException: Can't get master address from ZooKeeper; znode data == null 2023-04-15 02:56:27,380 INFO [M:0;jenkins-hbase20:43619] assignment.AssignmentManager(315): Stopping assignment manager 2023-04-15 02:56:27,380 INFO [M:0;jenkins-hbase20:43619] region.MasterRegion(167): Closing local region {ENCODED => 1595e783b53d99cd5eef43b6debb2682, NAME => 'master:store,,1.1595e783b53d99cd5eef43b6debb2682.', STARTKEY => '', ENDKEY => ''}, isAbort=false 2023-04-15 02:56:27,380 DEBUG [M:0;jenkins-hbase20:43619] regionserver.HRegion(1604): Closing 1595e783b53d99cd5eef43b6debb2682, disabling compactions & flushes 2023-04-15 02:56:27,380 INFO [M:0;jenkins-hbase20:43619] regionserver.HRegion(1626): Closing region master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-04-15 02:56:27,380 DEBUG [M:0;jenkins-hbase20:43619] regionserver.HRegion(1647): Waiting without time limit for close lock on master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-04-15 02:56:27,380 DEBUG [M:0;jenkins-hbase20:43619] regionserver.HRegion(1714): Acquired close lock on master:store,,1.1595e783b53d99cd5eef43b6debb2682. after waiting 0 ms 2023-04-15 02:56:27,380 DEBUG [M:0;jenkins-hbase20:43619] regionserver.HRegion(1724): Updates disabled for region master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-04-15 02:56:27,381 INFO [M:0;jenkins-hbase20:43619] regionserver.HRegion(2745): Flushing 1595e783b53d99cd5eef43b6debb2682 1/1 column families, dataSize=345.13 KB heapSize=393.82 KB 2023-04-15 02:56:27,478 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-04-15 02:56:27,478 INFO [RS:0;jenkins-hbase20:34681] regionserver.HRegionServer(1228): Exiting; stopping=jenkins-hbase20.apache.org,34681,1681527209133; zookeeper connection closed. 2023-04-15 02:56:27,478 ERROR [Listener at localhost.localdomain/33693-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@4e6e588f rejected from java.util.concurrent.ThreadPoolExecutor@41874b2e[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 2335] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-04-15 02:56:27,480 INFO [Shutdown of org.apache.hadoop.hbase.fs.HFileSystem@64447cc1] hbase.MiniHBaseCluster$SingleFileSystemShutdownThread(215): Hook closing fs=org.apache.hadoop.hbase.fs.HFileSystem@64447cc1 2023-04-15 02:56:27,480 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): regionserver:34681-0x100b28b0e4a0001, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-04-15 02:56:27,480 ERROR [Listener at localhost.localdomain/33693-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@1862784 rejected from java.util.concurrent.ThreadPoolExecutor@41874b2e[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 2335] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.hadoop.hbase.zookeeper.PendingWatcher.process(PendingWatcher.java:38) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-04-15 02:56:27,481 INFO [Listener at localhost.localdomain/33693] util.JVMClusterUtil(335): Shutdown of 1 master(s) and 1 regionserver(s) complete 2023-04-15 02:56:27,792 INFO [M:0;jenkins-hbase20:43619] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=345.13 KB at sequenceid=578 (bloomFilter=true), to=hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/.tmp/proc/fb216720e2f94d62a103cb01f12f7fa1 2023-04-15 02:56:27,797 INFO [M:0;jenkins-hbase20:43619] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for fb216720e2f94d62a103cb01f12f7fa1 2023-04-15 02:56:27,798 DEBUG [M:0;jenkins-hbase20:43619] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/.tmp/proc/fb216720e2f94d62a103cb01f12f7fa1 as hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/proc/fb216720e2f94d62a103cb01f12f7fa1 2023-04-15 02:56:27,801 INFO [M:0;jenkins-hbase20:43619] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for fb216720e2f94d62a103cb01f12f7fa1 2023-04-15 02:56:27,801 INFO [M:0;jenkins-hbase20:43619] regionserver.HStore(1080): Added hdfs://localhost.localdomain:34069/user/jenkins/test-data/a1e5f3f7-2798-46ce-0467-ddc6831bd2a5/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/proc/fb216720e2f94d62a103cb01f12f7fa1, entries=65, sequenceid=578, filesize=24.7 K 2023-04-15 02:56:27,801 INFO [M:0;jenkins-hbase20:43619] regionserver.HRegion(2948): Finished flush of dataSize ~345.13 KB/353417, heapSize ~393.80 KB/403256, currentSize=0 B/0 for 1595e783b53d99cd5eef43b6debb2682 in 421ms, sequenceid=578, compaction requested=false 2023-04-15 02:56:27,803 INFO [M:0;jenkins-hbase20:43619] regionserver.HRegion(1838): Closed master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-04-15 02:56:27,803 DEBUG [M:0;jenkins-hbase20:43619] regionserver.HRegion(1558): Region close journal for 1595e783b53d99cd5eef43b6debb2682: 2023-04-15 02:56:27,805 INFO [master:store-WAL-Roller] wal.AbstractWALRoller(243): LogRoller exiting. 2023-04-15 02:56:27,805 INFO [M:0;jenkins-hbase20:43619] flush.MasterFlushTableProcedureManager(83): stop: server shutting down. 2023-04-15 02:56:27,806 INFO [M:0;jenkins-hbase20:43619] ipc.NettyRpcServer(158): Stopping server on /148.251.75.209:43619 2023-04-15 02:56:27,807 DEBUG [M:0;jenkins-hbase20:43619] zookeeper.RecoverableZooKeeper(172): Node /hbase/rs/jenkins-hbase20.apache.org,43619,1681527208212 already deleted, retry=false 2023-04-15 02:56:27,909 INFO [M:0;jenkins-hbase20:43619] regionserver.HRegionServer(1228): Exiting; stopping=jenkins-hbase20.apache.org,43619,1681527208212; zookeeper connection closed. 2023-04-15 02:56:27,909 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-04-15 02:56:27,909 ERROR [Listener at localhost.localdomain/33693-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@743fbab4 rejected from java.util.concurrent.ThreadPoolExecutor@652e2a7a[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 3906] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.hadoop.hbase.zookeeper.PendingWatcher.process(PendingWatcher.java:38) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-04-15 02:56:27,909 DEBUG [Listener at localhost.localdomain/33693-EventThread] zookeeper.ZKWatcher(600): master:43619-0x100b28b0e4a0000, quorum=127.0.0.1:51762, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-04-15 02:56:27,909 ERROR [Listener at localhost.localdomain/33693-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@63a0d466 rejected from java.util.concurrent.ThreadPoolExecutor@652e2a7a[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 3906] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-04-15 02:56:27,912 WARN [Listener at localhost.localdomain/33693] datanode.DirectoryScanner(534): DirectoryScanner: shutdown has been called 2023-04-15 02:56:27,918 INFO [Listener at localhost.localdomain/33693] log.Slf4jLog(67): Stopped HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:0 2023-04-15 02:56:28,031 WARN [BP-1118687208-148.251.75.209-1681527205747 heartbeating to localhost.localdomain/127.0.0.1:34069] datanode.IncrementalBlockReportManager(160): IncrementalBlockReportManager interrupted 2023-04-15 02:56:28,031 WARN [BP-1118687208-148.251.75.209-1681527205747 heartbeating to localhost.localdomain/127.0.0.1:34069] datanode.BPServiceActor(857): Ending block pool service for: Block pool BP-1118687208-148.251.75.209-1681527205747 (Datanode Uuid 9228a71f-02f5-4551-b279-c9b574701634) service to localhost.localdomain/127.0.0.1:34069 2023-04-15 02:56:28,035 WARN [refreshUsed-/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/138098eb-2451-2962-d451-8fb43de970f3/cluster_bb59e7f6-88db-fd92-3eb8-75ab1f712feb/dfs/data/data1/current/BP-1118687208-148.251.75.209-1681527205747] fs.CachingGetSpaceUsed$RefreshThread(183): Thread Interrupted waiting to refresh disk information: sleep interrupted 2023-04-15 02:56:28,035 WARN [refreshUsed-/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/138098eb-2451-2962-d451-8fb43de970f3/cluster_bb59e7f6-88db-fd92-3eb8-75ab1f712feb/dfs/data/data2/current/BP-1118687208-148.251.75.209-1681527205747] fs.CachingGetSpaceUsed$RefreshThread(183): Thread Interrupted waiting to refresh disk information: sleep interrupted 2023-04-15 02:56:28,067 INFO [Listener at localhost.localdomain/33693] log.Slf4jLog(67): Stopped HttpServer2$SelectChannelConnectorWithSafeStartup@localhost.localdomain:0 2023-04-15 02:56:28,223 INFO [Listener at localhost.localdomain/33693] zookeeper.MiniZooKeeperCluster(344): Shutdown MiniZK cluster with all ZK servers 2023-04-15 02:56:28,344 DEBUG [HBase-Metrics2-1] impl.GlobalMetricRegistriesAdapter(152): Removing adapter for the MetricRegistry: RegionServer,sub=Coprocessor.Region.CP_org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint 2023-04-15 02:56:28,359 INFO [Listener at localhost.localdomain/33693] hbase.HBaseTestingUtility(1293): Minicluster is down